Check for sample ratio mismatch before trusting an A/B test
Short answer: Before you trust an A/B test result, check that each variant received roughly the share of visitors you planned. If you set a 50/50 split and one side got noticeably more users than chance allows, you have a sample ratio mismatch (SRM), and the result is probably biased no matter how good the uplift looks. Find and fix the cause, usually tracking or redirect problems, then rerun the test rather than reading the numbers.
What sample ratio mismatch means
When you run an experiment, your testing tool assigns each visitor to a variant using the ratio you chose, for example 50% to the current checkout and 50% to the new one. With enough visitors, the counts should land close to that split. They won't be exactly equal, because assignment is random, but a statistical test can tell you whether the gap is bigger than random chance would explain.
A sample ratio mismatch is when it is. Experimentation platforms treat it as a red flag: GrowthBook's troubleshooting guide describes an SRM error as a clue that traffic to the experiment isn't balanced, and Statsig's monitoring checks include "exposures balanced with no sample ratio mismatch" as one of the health checks to review after launch and again before reading results.
Why it matters more than the uplift
An SRM usually means some visitors were lost or counted differently in one variant. Those missing visitors are rarely a random slice. Typical examples:
- The new variant loads slower, so impatient visitors leave before the tracking event fires. Only the patient ones are counted, and they convert better anyway.
- A redirect test sends variant B to a new URL, and some browsers or ad blockers drop the tracking on the redirected page.
- A bug crashes the new variant on one browser, so those users vanish from the data.
In each case, the variant with fewer visitors isn't being compared fairly. The test may show a winner that doesn't exist, or hide a real loss. Teams that ship an SRM-affected "winner" often see the improvement evaporate after launch.
How to check for it
Most testing platforms run the check for you and show a warning. If yours doesn't, or you analyse data yourself:
- Count the users assigned to each variant, using the same unit you randomised on (users, not page views, if you assign per user).
- Compare the counts with the expected ratio using a chi-squared goodness-of-fit test. Free online calculators and every statistics library offer one.
- Treat a very small p-value as a mismatch. Many teams use a strict threshold for this check, because it's meant to catch broken experiments, not subtle effects.
Check early, a day or two after launch, so you don't waste weeks of traffic on a broken setup. Check again before you make a decision.
Common causes and where to look
Tracking fires at different times. If the "exposure" event is sent after something that differs between variants, such as a heavier page or an extra script, one variant loses more visitors before they're counted. Fire exposure as early and consistently as possible.
Redirect tests. Split-URL tests are especially prone to SRM because the redirect adds a step where visitors can drop off and tracking can break. Server-side assignment avoids the extra hop.
Bots and filters. If bot filtering, internal-traffic exclusion or consent handling treats variants differently, counts drift apart. Check that filters apply equally to both.
Assignment bugs. Hashing on an ID that changes mid-visit, or caching a page with one variant baked in, can push more people into one group.
Variant-specific errors. Look at error logs and browser breakdowns per variant. A crash in one browser for one variant is a common culprit.
Mid-test changes. Changing the traffic split or targeting while the test runs can produce imbalances depending on how your tool handles it.
What to do when you find one
- Don't read the results. Not even "directionally". The data is compromised.
- Find the cause using the list above. Segment the counts by browser, device, country and traffic source to see where the imbalance lives.
- Fix it and restart the experiment with fresh data.
- Add an automatic check so future tests warn you within the first day.
Preventing it in the next test
A few habits stop most mismatches before they start:
- Run an A/A test once when you set up a new tool: both groups see the same page. If even that shows a mismatch, the problem is your tracking, not your design.
- Assign on the server where you can, so the split happens before any page weight or script differences come into play.
- Use one stable ID per visitor for both assignment and analysis.
- Freeze the setup once the test starts. If something must change, restart with a new experiment.
- Write the expected split down in the test plan, so whoever analyses it later checks against the right numbers.
A worked example
Imagine a hypothetical Christchurch outdoor gear store testing a new product page layout with a 50/50 split. After a week, the new layout showed a much higher add-to-cart rate. The team's tool also flagged an SRM: the new variant had clearly fewer visitors than expected. Segmenting showed the gap was almost entirely on one mobile browser, where the new page's image carousel threw an error before the tracking tag loaded. They fixed the error, restarted the test, and the new layout's advantage shrank to a small, steady gain.
FAQ
How big a gap counts as a mismatch?
It depends on sample size. A small percentage gap can be significant with large traffic. Use a statistical test rather than eyeballing it.
Can I just reweight the results?
No. Reweighting assumes the missing visitors were random, which is rarely true. Fix the cause and rerun.
Does SRM apply to uneven splits too?
Yes. If you planned 90/10, check against 90/10.
My tool shows no SRM warning. Am I safe?
You're safer, but still check that the tool counts the same unit you randomised on, and that your own analysis uses the same filters.
If your A/B tests give results you don't quite trust, tell us how your experiments are set up and we'll help you check them.