A/B testing

Testing only works if you have enough visitors. Otherwise you're measuring noise.

HypothesisSet-upReliabilityRoll-out
Analysis & optimisation at Flii Media

How a test runs

When testing isn't possible

With a hundred visitors a month, a reliable test takes longer than the subject stays relevant. Then testing is pointless, and you're better off relying on design principles and user interviews. We'd rather tell you that up front.

Stopping early is the biggest pitfall

After three days variant B is ahead, so it wins. Two days later it has flipped. If you stop as soon as you like the outcome, you roll out noise as an improvement. The duration is fixed in advance and we don't deviate from it.

Test what matters

A button colour rarely delivers more than one per cent. A different offer, a shorter form step or a different headline can make a difference of tens of per cent. Start with the big things.

A question or project on this topic?

Cases using this service

Royal HaskoningDHV→ For one of the world's largest engineering firms, we started with a targeted Google Ads campaign… Campaigns · Search engines · Tracking Soest Machinery→ Soest Machinery is a Dutch manufacturer of industrial finishing equipment, including machines… Search engines · Development · Tracking Stichting De Buurt→ Stichting De Buurt works towards neighbourhoods where people greet, know and help each other. Through initiatives… Campaigns · Social media · Tracking