A/B Testing for Online Stores: When It Pays Off and How to Set It Up Right
What an A/B test is, how many visits it needs and what to do when your traffic is not enough. With a calculator for your store.

Jason Spizert
Updated on October 21, 2026
Published 5 min read
The short answer
An A/B test shows two groups of visitors two variants of the same page and measures which one sells more. It only pays off if your store has enough visits and purchases: at a 1% conversion rate, a test needs about 79,000 visits to reliably detect a 20% difference. If your traffic is not enough, a before-and-after comparison over fixed time periods is the better choice.
A/B tests are considered the standard of conversion optimization. That is true, with one condition people rarely say out loud: a test needs volume. Without enough purchases, it measures chance, and you decide on a number that would look different in the next test.
What is an A/B test?
In an A/B test, also called a split test, a tool splits your visitors into two groups at random: A sees the current page, B a changed one. Both run at the same time under the same conditions. That is exactly what sets it apart from a before-and-after comparison:
Gut feeling vs. data
Almost every store has this discussion: “The new page looks so much better.” In the end, whoever is loudest or controls the most budget decides. Everyone is sure of their opinion, and nobody asks the customers.
What a test replaces
A clean A/B test replaces “I think” with “customers have shown us.” A badly calculated one replaces an opinion with chance, and that is more dangerous, because it looks like proof.
What statistical significance means
Significance answers one single question: how likely is it that a difference is just chance? The easiest way to see it is with a coin.
A result usually counts as significant if a gap like this would occur in fewer than 5 out of 100 cases when both variants are equally good. That is the 95% you see in testing tools. But significant does not mean important: a tiny difference can be significant and still not be worth it.

When an A/B test pays off
How many visits a test needs depends on your conversion rate and on the difference you want to detect. The smaller both are, the more visits you need. Calculate it for your store:
Many stores sit in the low single-digit percentage range, and small changes like a new button color rarely move the result by more than a few percent. Big changes, such as a rebuilt product page, have a better chance of becoming visible:
At what store size testing pays off
This leads to an uncomfortable truth: for small stores, conversion optimization through A/B tests does not pay off. Two stores, both with a €50 order value and a 1.5% conversion rate:
The small store waits four months for an extra €2,000 a month, and in that time season, advertising and product range all change. It gets more from a clear offer, more of the right visitors and fixing obvious breaks in the store. The big store knows after just under a month whether a change brings in €20,000 a month. For that store, testing is the fastest lever there is.
How to set up an A/B test correctly
The hypothesis is the most important part. It says what you change, what should happen and why. Where good hypotheses come from is covered in our other guides.
The most common mistakes
Most tests do not fail because of the tool, but because of one of these four points:
What if your traffic is not enough?
Then an A/B test is the wrong tool, and that is nothing to be ashamed of. The more honest path has three steps. How we approach it is described on our page about conversion optimization.
A/B testing tools
Which tool fits depends mainly on what you want to test and how cleanly it works with your tracking.
We record every test in our own CRO cockpit: hypothesis, duration, result and what we learn from it. That way, even a test without a winner is not wasted.

Frequently asked questions
In an A/B test, randomly selected visitors see two variants of the same page: A is the current one, B the changed one. You measure which variant brings more purchases or revenue. Because both groups run at the same time under the same conditions, the difference can be traced back to the change.
Until the number of visits you calculated in advance is reached, and for at least two full weeks so every weekday is included twice. If you stop as soon as one variant is ahead, you often get a result that does not hold up later.
It depends on your conversion rate and on the difference you want to detect. At a 1% conversion rate, a test needs about 79,000 visits to reliably detect a 20% difference. The calculator in this article shows the number for your store.
It depends on visits and orders, not on revenue alone. As a rough guide: a store with €10,000 in monthly revenue and a €50 order value needs about four months to reliably detect a 20% improvement. A store with €200,000 detects 10% in just under a month. Below that, your offer, your traffic and obvious breaks in the store do more than tests.
It describes how unlikely it is that the measured difference is just chance. 95% is the usual threshold. But significance says nothing about whether the difference is big enough to pay off commercially.
A/B test and split test usually mean the same thing: two variants against each other. A multivariate test tests several elements at once in combinations. It needs far more traffic and does not pay off for most stores.
Yes, with apps or external testing tools. Which one fits depends on what you want to test: page content, themes or prices. More important than the tool is that tracking, duration and goal metric are fixed in advance.









