A/B test significance calculator
Did B win?Or get lucky?
Enter the visitors and conversions for each version, and this A/B test calculator tells you whether the difference is statistically significant. It runs a two-proportion z-test and gives the p-value, the confidence, the uplift with its confidence interval and a verdict in plain English. Add your daily visitors and it estimates how many more days the test needs. Below it, every number is explained, along with when to stop a test and what a “no difference” result means.
By the Builderify team
If A and B really converted the same, a gap at least this big would turn up only 2.7% of the time. That is under the 5% you allowed at 95% confidence, so it is unlikely to be luck alone. The true lift is most likely between +2.6% and +56.3%.
Keep it running about 11 more days
To catch a 26.7% lift from a 3.00% baseline 8 times in 10, at 95% confidence, a test needs about 8,052 visitors per version. Each has at least 5,000. At 600 visitors a day, that is about 11 more days.
How to read an
A/B test result.
Number by number.
Look at the interval before the uplift. The uplift is the single best guess; the interval is how far off that guess could be. In the example, a +26.7% lift with an interval of +2.6% to +56.3% means B is probably better, by somewhere between a little and a lot.
A p-value says how surprising your data would be if there were no difference. The American Statistical Association’s statement on p-values is blunt about the two common misreadings: a p-value does not measure the probability that the hypothesis is true, and it does not measure the size of an effect or the importance of a result.ASA
| Number | Your test | What it tells you |
|---|---|---|
| Conversion rate, A and B | 3.00% · 3.80% | Conversions divided by visitors for each version. Everything else is built on these two numbers. |
| Difference | +0.80 pts | B’s rate minus A’s, in percentage points. The absolute change: 3.0% to 3.8% is +0.8 points. |
| Relative uplift | +26.7% | The difference as a share of A’s rate: +0.8 points on 3.0% is +26.7%. It is the number vendors put in headlines, and on its own it says nothing about luck. |
| z-score | +2.21 | How many standard errors apart the two rates are. Beyond ±1.96 is significant at 95%, two-sided. |
| p-value | 0.0273 | If A and B truly converted the same, how often a gap at least this big would turn up by chance. It is not the chance that B is better, and it is not the size of the effect. |
| Confidence | 97.27% | One minus the p-value, the way most calculators show it. The result is significant when it reaches the level you chose. |
| Interval for the difference | +0.09 to +1.51 pts | The range of true differences your data fit, at the confidence you chose. If it crosses zero, the test is not significant. |
| Interval for the uplift | +2.6% to +56.3% | The same range as a share of A’s rate. Read the low end: it is the smallest lift your data still support. |
| Chance B beats A | 98.6% | A Bayesian reading of the same counts: how likely B’s true rate is above A’s. It is the figure Builderify’s A/B Testing calls a winner on. |
| Visitors still needed | Keep it running about 11 more days | The sample a lift the size you see (or 5%, if it is smaller) needs to be caught 8 times in 10, turned into days at your traffic and never less than a full week. Chapter 04 explains it. |
The middle column follows whatever you enter in the calculator. The example is made up: 5,000 visitors on each version, 150 conversions on A and 190 on B, 600 visitors a day.
One-sided or two-sided
A/B test?
Two, unless you decided first.
A two-sided test asks “is B different from A?”; a one-sided test asks only “is B better?”. At 95%, a two-sided test splits its 5% between the two directions, 2.5% each, while a one-sided test spends all 5% on one.UCLA So the one-sided test calls a lift significant with a smaller gap, and when B is ahead the same data give it half the p-value.
The price is blindness in the other direction: a one-sided test can’t tell you B is worse. UCLA’s statistics FAQ says a one-sided test is only appropriate when missing an effect the other way would be negligible, and that choosing one to reach significance, or after a two-sided test came up short, is not.UCLA For a website, a version that quietly loses sales is rarely negligible, so this calculator and Builderify both default to two-sided.
| Confidence | Two-sided: |z| beyond | One-sided: z beyond |
|---|---|---|
| 90% | 1.645 (5.0% in each tail) | 1.282 (10% in one tail) |
| 95% | 1.960 (2.5% in each tail) | 1.645 (5% in one tail) |
| 99% | 2.576 (0.5% in each tail) | 2.326 (1% in one tail) |
Worked out on this page from the inverse of the normal distribution. Switch the calculator to one-sided and the example’s p-value halves, from 0.0273 to 0.0137.
Which confidence level?
95% is the convention, and it is what Builderify uses. Drop to 90% only for a cheap change you could undo tomorrow; go to 99% when a wrong call is expensive, knowing the test will need more visitors. Pick the level before the test starts, not after you see the number.
Can I stop an A/B test
once it’s significant?
Not the moment it turns green.
A significance test is only valid if you look once, at a sample size you chose in advance. Check every day and stop the first time it crosses 95%, and you are no longer running a 5% test. Evan Miller’s classic example is a change that does nothing, checked after every visitor and stopped at the first significant reading: it is called a winner 26.1% of the time, “more than five times” the 5% you thought you had.Evan Miller
| You peeked | For a real 5%, the result must reach |
|---|---|
| 1 time | 2.9% |
| 2 times | 2.2% |
| 3 times | 1.8% |
| 5 times | 1.4% |
| 10 times | 1.0% |
From Evan Miller’s article: peek ten times, and a result has to show 1% significance to really mean 5%.Evan Miller
-
01
Decide the sample before you start
Pick the smallest lift worth having, work out the visitors it needs, and read the result when you get there. Miller’s own advice is that committing to a sample size in advance removes the problem.Evan Miller
-
02
Run whole weeks
Weekday and weekend visitors behave differently, so a test that ends on a Tuesday over-counts some days. Builderify won’t let you declare a winner before a test has run seven days, for that reason.
-
03
Look all you like, but don’t act on it
Watching the dashboard is harmless. Stopping because it looks good is the problem. If you must check early, use the stricter levels in the table, or a method built for repeated looks.
-
04
Treat a huge early lift with suspicion
With 80 visitors a version, one or two lucky conversions can move the uplift by 100%. That is why the calculator, like Builderify, won’t give a verdict until each version has 100 visitors and 10 conversions.
How long should an
A/B test run?
Until the sample is in.
Long enough to reach the visitors a lift of the size you care about needs, and never less than a full week. The calculator’s “keep it running” line does that sum for the lift it sees: in the example, catching a 26.7% lift from a 3% baseline 8 times in 10 needs about 8,052 visitors per version, and at 600 visitors a day the test is 11 days short.
Significant now, but short of the sample
The example is significant at 95% and still gets “keep it running”. That is the peeking problem from chapter 03 in numbers: the test reached significance before it reached the sample a lift that size needs, so the result could still fade. If you planned 8,052 a version, wait for 8,052.
Small differences take a very long time
The visitors needed grow with the square of how small the lift is: halve the lift and you need roughly four times the traffic. When the estimate passes 90 days the calculator says so, because a difference that small may never show at your traffic. Test a bolder change instead: a new offer or headline, not a new shade of button.
Planning a test from scratch
If you don’t know your baseline yet, start with what a good conversion rate is for your kind of page. Each platform guide also has a planner for the test before it starts: the Shopify A/B testing guide has a sample-size planner, the WordPress A/B testing guide works out how many weeks a test needs, the Wix and Webflow guide estimates how long yours will take, and the Squarespace A/B testing guide pairs a significance check with a sample-size table.
A/B test not significant?
Read the interval, then decide.
“Not significant” means this test couldn’t tell the versions apart, not that they are the same. Altman and Bland made the point in the BMJ under a title that says it all, “Absence of evidence is not evidence of absence”, and recommended looking at the confidence interval rather than just the p-value.BMJ The interval tells you which of three situations you are in.
Narrow, around zero
For example −2% to +3%. The change made little difference either way. Keep whichever version is simpler or cheaper to run, and spend your traffic on a bigger idea.
Wide, mostly positive
For example −5% to +40%. B may well be better, but the test is too small to say. Keep it running if the calculator gives a sensible number of days; otherwise call it unproven, not a loss.
Wide, both ways
For example −30% to +35%. The test learned almost nothing. Most often there wasn’t enough traffic for a change this subtle. Test something bolder, or a page with more visitors.
Two things not to do: don’t switch to one-sided after the fact to get under the line, and don’t start slicing the result by device or source until one slice “wins”. Each extra comparison is another chance of a lucky winner. If too few people convert to test anything at all, work on capture first; an exit-intent popup from Popups gives leaving visitors a smaller next step.
How this A/B test
calculator works.
Nothing hidden.
It is a pooled two-proportion z-test, the standard test for comparing two conversion rates. The test pools both versions to get the standard error, because it assumes no difference; the confidence interval uses each version’s own rate, which is the textbook pairing.Penn State
The normal distribution is computed with a double-precision approximation, checked against Python’s statistics library to within one part in a hundred million. The first result on this page is worked out on the server with the same formulas the calculator runs in your browser, and the two were tested against each other on cases including equal rates, zero conversions and 100% conversion.
The test
p-value: two-sided 2 × Φ(−|z|), one-sided Φ(−z), where Φ is the normal distribution. Significant when p is below 1 minus your confidence level.
The intervals
z* is the critical value from chapter 02. One-sided, only the lower bound is shown. The uplift interval needs a conversion on each side.
Chance B beats A
A normal approximation of each posterior, which is accurate at the 100 visitors and 10 conversions a verdict needs anyway.
Visitors still needed
For 80% power: p1 is A’s rate and p2 is A’s rate moved by the lift you see (at least 5%). Never below 100 visitors and 10 expected conversions per version. Days = the shortfall per version ÷ half your daily visitors, and at least enough to make up seven days in all.
Only two versions at a time. With more variants against one control, each extra comparison is another chance of a lucky winner, and the bar has to rise; Builderify raises it for you.
An A/B test that won’t
call a winner too early.
The same rules, built in.
A/B Testing does these sums for you, and holds back until they can be trusted. You change text, markup or visibility in a visual editor, split traffic by URL rule and measure goals: a page visit, a click on an element, or a custom event. The results apply the guard rails from this page.
Nothing until 100 and 10
Each variant needs at least 100 visitors and 10 conversions before a result is reported. Until then the test reads “collecting”, with a note saying what is still missing.
A full week before a winner
A winner can only be declared after the test has run seven days, because weekday and weekend visitors behave differently and an early lead often reverses.
Called on the chance to win
A variant is called when it has at least a 95% chance to beat the control on the primary goal. With several variants the bar rises for each one, so extra variants don’t produce lucky winners.
The z-test in the details
Each result’s details show the two-proportion z-test confidence, a 95% interval for the uplift and the expected loss if you pick that variant.
About N more days
An undecided test gets a rough “about N more days at this pace”, worked out the way this calculator does it, from the traffic the test has had so far.
A broken split is flagged
If visitors split far from the weights you set, the results warn you and no winner can be declared, because a lopsided split usually means visitors are being lost or double counted on one side.
Before you launch, it estimates the wait. The editor shows how many visitors each version needs to spot a 20% lift on a typical 3% conversion rate and, using your site’s recent daily traffic from User Journey, roughly how long that will take. So a test that would need half a year is obvious before it starts.
Setting it up on your site. Builderify installs with one script tag. The platform guides walk through it: A/B testing on WordPress without breaking your cache, on Shopify next to Rollouts, on Squarespace through Code Injection, and on Wix or Webflow.
Launching a test costs 25 credits, once, and results cost 1 credit per 1,000 exposures and conversions recorded. Every account gets 250 free credits a month, and one-time packs start at $9 with no monthly fee. See pricing.
Start freeA/B test significance, before you ask.
Compare the two conversion rates with a two-proportion z-test. Pool both versions' conversions to get the standard error, divide the difference in rates by it to get z, and turn z into a p-value with the normal distribution. If the p-value is below 0.05, the result is significant at 95% confidence, two-sided. The calculator on this page does the sums and adds a confidence interval.
Two-sided, unless you decided before the test that a worse version B would change nothing you do. A one-sided test puts all of its 5% on one direction, so it calls a lift significant sooner, but it cannot tell you B is worse. Switching to one-sided after seeing the data, to get under 0.05, is not valid.
No. Checking again and again and stopping at the first significant result inflates false positives: in Evan Miller's worst case, checking after every visitor, a change that did nothing came out significant 26.1% of the time instead of 5%. Decide the sample size before you start, run whole weeks, and read the result once the sample is in.
That this test could not tell the versions apart, not that they are the same. Look at the confidence interval: if it is narrow around zero, the change made little difference either way and you can keep whichever version is simpler. If it is wide, the test was too small to judge, so run it longer or test a bolder change.
No result is reported until each variant has at least 100 visitors and 10 conversions, and a winner can only be declared after a full week. A variant is then called when it has at least a 95% chance to beat the control on the primary goal, a bar that rises with each extra variant, and the details show the two-proportion z-test and a 95% interval for the uplift. Launching a test costs 25 credits, and results cost 1 credit per 1,000.
Sources
- Evan Miller How Not To Run an A/B Test (18 April 2010)
- American Statistical Association Wasserstein and Lazar, The ASA’s Statement on p-Values: Context, Process, and Purpose, The American Statistician, 2016
- UCLA OARC FAQ: What are the differences between one-tailed and two-tailed tests?
- Penn State STAT 200, 9.1.2.1: Normal Approximation Method Formulas (two proportions)
- The BMJ Altman and Bland, Statistics notes: Absence of evidence is not evidence of absence, BMJ 1995;311:485
Sources were read on 29 October 2026. How Builderify’s A/B Testing reports a result (the minimums, the week, the chance to beat the control, the z-test in the details and the “more days” estimate) comes from Builderify’s own code; the prices come from the rate card on this site. The calculator is a guide to reading a test, not a substitute for planning one.
Stop losing the visitors
you already paid for.
More leads from the traffic you already have, and follow-up that sends itself. Free to start. No card, no monthly fee.