Sample Ratio Mismatch

What Is Sample Ratio Mismatch? Meaning & Examples

Sample ratio mismatch is a statistically significant discrepancy between the expected and observed sizes of experiment groups. In other words, sample ratio mismatch occurs when user allocation deviates from expected ratios. You set up treatment and control groups with a defined traffic distribution, say 50/50, and the actual count of users assigned to each variant drifts far from that target.

To build intuition, think of a fair coin. Flip it 100 times, and you might see 52 heads and 48 tails. That is normal variation. But if you flip it 1,000 times and see 700 heads, something is clearly off with the coin. The same logic applies when you randomly assign users in an experiment.

Here is a concrete numeric example. You launch a test with a planned 50/50 split and 100,000 total users. You expect roughly 50,000 in each group. Instead, you observe 63,000 in control and 37,000 in treatment. That gap is far too large for random sampling noise, and it triggers an SRM alarm.

The term sample ratio mismatch SRM is widely used in the context of online controlled experiments, feature tests, and marketing experiments. Whether you are testing a new checkout page layout or a pricing change, the principle is the same: if SRM occurs when user assignment deviates from the expected ratio, the experiment has a problem worth investigating.

Why sample ratio mismatch matters

Random assignment is the foundation of trustworthy A/B testing. Statistical tests assume random assignment to groups, and when that assumption breaks, every downstream metric becomes suspect. Sample ratio mismatch (SRM) can lead to misleading conclusions in A/B testing results, and in the worst cases, it can cause the entire experiment to produce invalid results.

When an SRM issue surfaces, it usually signals selection bias, where certain types of users are more likely to end up in one group than another. This means the groups are no longer comparable, and the apparent treatment effect may actually be due to differences in who saw each variant, not the effect of the treatment itself. SRM indicates potential biases in user assignment or data processing that have nothing to do with the treatment being tested.

Consider a common scenario: you run an experiment on a checkout page to test a new design intended to increase revenue. Due to a targeting bug, a disproportionate share of high-value mobile users is routed into the treatment group. As a result, revenue metrics spike in the treatment group. However, this lift is not caused by the new design but by the uneven distribution of users. Without detecting SRM, your team may mistakenly believe the new design is effective and roll it out to all users, only to find no real improvement in revenue.

The impact of SRM is significant and widespread in the industry. Research from Microsoft shows that about 6 percent of A/B tests experience SRM, while LinkedIn reports rates near 10 percent for more complex experiments. This prevalence highlights the importance of routinely checking for SRM to maintain experiment integrity. Experiments affected by SRM tend to show twice as many statistically significant metrics compared to clean experiments. This inflation of false positives means that SRM not only undermines trust in individual tests but can also distort the overall experimentation program by promoting false discoveries.

Even small allocation differences can have outsized effects. For example, a 1 percent deviation from the intended sample ratio can distort key metrics by as much as 2 percent. When experiments are run at scale across many teams and products, these small inaccuracies compound, leading to widespread incorrect conclusions about what works and what does not.

SRM also complicates the interpretation of experiment results because it violates the assumption that treatment and control groups are comparable. This comparability is essential for causal inference. If the groups differ systematically in characteristics like device type, geography, or user behavior, then any observed differences in outcomes may reflect these underlying differences rather than the effect of the tested change. For example, if more engaged users disproportionately fall into one group, that group may appear to perform better regardless of the treatment.

Another consequence of SRM is that it reduces the statistical power of the experiment. When group sizes deviate from the planned ratio, the effective sample size for detecting true effects is reduced. This reduction can increase the likelihood of type II errors, where a real effect goes undetected. Conversely, SRM can inflate false positive rates by introducing bias that mimics a treatment effect.

SRM can also lead to wasted resources and lost time. Experiments affected by SRM often need to be stopped and restarted after the root cause is identified and fixed. This delay slows down the product development cycle and reduces the overall velocity of experimentation. Teams may also lose confidence in the experimentation platform if SRM issues occur frequently, undermining the culture of data-driven decision-making.

Detecting SRM early is crucial to minimize its impact. Automated monitoring tools that track the allocation ratios and alert experimenters when deviations exceed thresholds help prevent prolonged exposure to biased data. These tools enable quick investigation and remediation, preserving the integrity of ongoing tests.

In addition to detection, understanding the root causes of SRM is vital. Common causes include bugs in randomization code, inconsistent user identifiers, targeting errors that exclude users unevenly, and data logging issues such as dropped events or delayed ingestion. Each cause requires a different approach to resolution, emphasizing the need for thorough diagnostics.

Finally, SRM awareness encourages better experiment design and implementation practices. Teams that prioritize avoiding SRM invest in robust randomization libraries, consistent user ID management, and comprehensive logging pipelines. They also establish clear protocols for monitoring and responding to allocation imbalances, which collectively improve the quality and reliability of their experimentation efforts.

How sample ratio mismatch works

Every experiment starts with an intended allocation rule. For example, "assign each new user to control or treatment with 50 percent probability." This defines the expected sample ratio. In a correctly implemented test, the realized counts fluctuate around this rule, but those deviations should be consistent with chance variation for the given sample size.

SRM detection compares observed group sizes with expected sizes under the planned sample ratio and evaluates whether the difference is unusually large. The evaluation uses statistical methods that quantify how surprising the observed split would be if randomization were working perfectly.

This comparison must happen at the level at which randomization happens. If your platform randomizes at the user level, SRM detection should count unique users, not sessions or page views. If you use sessions as the unit of analysis when users are the randomization unit, behavioral differences between variants (like one variant encouraging more users to return) can create an apparent imbalance that is not actually a randomization failure.

Most mature experimentation platforms automate SRM detection. The system runs checks repeatedly while the test is live, catching problems within hours rather than only at the end. However, SRM does not tell you which group is "right" or what the true treatment effect is. It only flags that something about randomization or data collection is likely broken.

Diagnosing sample ratio mismatch is a two step process: first detect that an SRM exists, then investigate the root cause.

Detection

Detection relies on the chi squared test, which is the most commonly used approach. Chi-square tests are commonly used to identify SRM issues, and a chi-squared goodness-of-fit test can detect SRM reliably across a range of sample sizes. You compute expected counts per variant (total users times the planned allocation proportion), compare them to the observed counts, and calculate the test statistic. Chi-squared tests can identify sample ratio mismatch (SRM) by producing an SRM p value that tells you the probability of seeing a mismatch at least as extreme under correct randomization.

An SRM check fails if groups are not split 50/50 as intended (or whatever your target ratio is) and the resulting p value is below your threshold. An SRM p value below 0.01 indicates unbalanced exposures, and a p value below 0.01 is the threshold many platforms use, though some organizations choose 0.001 or even 0.0005 to further reduce false positives.

When a very low p value appears, for instance 10 to the power of negative 10, it is overwhelmingly unlikely that the mismatch is random.

Investigation

After confirming SRM, investigation begins by plotting traffic counts over time to identify when the mismatch started. Tying the onset to a specific code deploy, the experiment's start date, or a configuration change helps quickly identify the source.

Next, analyze segments by slicing aggregated data across platform, country, browser type, logged in status, or traffic source to localize where the imbalance is strongest.

Also compare triggered versus untriggered users for experiments that require a specific trigger, such as viewing a widget or reaching a certain page. If the trigger logic is variant sensitive, it can produce missing users in one group.

You can also apply permutation tests to validate the chi squared result on smaller data sets, or use Bayesian A/B testing methods for an alternative perspective on the mismatch probability.

Ratio mismatch can originate at several layers of the experiment stack. Here is a catalog of the most frequent root causes.

Assignment stage issues include bugs in the randomization code, incorrect bucketing logic, or two different systems assigning users with inconsistent rules, all of which can skew the split. SRM can occur due to incorrect user assignment in experiments when the treatment assignment mechanism is flawed. Incorrect bucketing can lead to sample ratio mismatch even when the hashing function appears correct at first glance.

User identifier problems such as user ID corruption undermine consistent group assignment. Cookie resets, shared devices in households, anonymous versus logged in state transitions, or duplicate IDs merging total traffic incorrectly are common culprits.

Targeting and eligibility errors happen when filters exclude some users from one variant after assignment, or feature flags are applied only to a subset of devices, creating an uneven distribution. If your test targets specific user segments but the targeting interacts with the variant code, one group may silently lose participants.

Execution problems include redirect loops, page load failures, or app crashes that happen more often in one variant preventing exposures from being recorded. This creates a set of missing users in the affected group. Uneven rollout timing can also cause sample ratio mismatch if one variant launches hours before the other.

Logging and data pipeline issues arise from tracking and logging failures. Dropped events from a particular platform, delayed data ingestion through stream processing systems, or aggressive bot filtering that removes more users from one group are all potential sources. Data processing errors can lead to sample ratio mismatch downstream, even when the assignment itself was clean.

Analysis stage errors include applying different filters to control groups and treatment groups, resetting the experiment's start date without resetting counts, or analyzing only users who completed a trigger influenced by the treatment, all of which can create the illusion of a ratio mismatch.

The best approach is to map potential causes to specific experiment components: assignment, exposure, logging, and analysis. This systematic framework, similar to knowledge discovery in data mining, helps narrow down the root cause efficiently.

Imagine you launch an A/B test with an intended 50/50 split. After three days, your dashboard shows 55 percent of users in treatment and 45 percent in control. That visual check is your first signal.

For a formal check, pull the unique user counts per variant from your experiment dashboard. Compute expected counts based on the planned allocation proportion. Run the chi squared test and compute the SRM p value. If the p value is below your threshold, you have a confirmed SRM. Spreadsheet tools can compute this directly using built in chi squared formulas, making this accessible for anyone, not just a data scientist.

Checks should always be performed on unique users assigned to each variant, not on sessions or page views, when the randomization unit is the user. Otherwise, performance differences between variants that change visit frequency could create a false alarm.

Mature experimentation platforms often run SRM checks automatically. For instance, Statsig automatically checks for SRM in every experiment, flagging exposures as balanced or unbalanced. These low latency checks process each exposure event batch so that problems are caught within hours. Real-time dashboards help monitor experiment exposures effectively, and monitoring tools can alert users to potential SRM issues in real time, catching problems that manual inspection would miss.

Plotting the SRM p value over time is especially useful. If the mismatch appeared only after a specific deployment, that timestamp becomes a strong lead for the investigation. Pair this view with user dashboards that show diagnostic tools like segment level breakdowns, and you can troubleshoot experiments much faster.

Examples of sample ratio mismatch

Here are a few concrete scenarios that illustrate different failure modes.

Marketing traffic exclusion

A paid campaign sends visitors to your site with a specific URL parameter. A code bug strips that parameter for one variant, and the experimentation platform fails to log those visitors. The result: a 60/40 observed split instead of 50/50, with more users appearing in control. A team trying to improve weekly revenue from that campaign would draw incorrect conclusions about the treatment's performance.

Performance regression

The treatment variant introduces heavier JavaScript that slows page load time. Some visitors abandon the page before the experiment exposure event fires. The underlying data shows many fewer logged treatment users, producing a severe ratio mismatch. The experiment results appear to show that the treatment has lower conversion, but the real story is that the slowest loading visitors never got counted at all.

Mid test bot detection change

Partway through the experiment, the engineering team updates bot detection rules. The new rules remove more high frequency users from the control group than from treatment. This creates an SRM and simultaneously biases revenue metrics upward for control, because the remaining control users happen to be higher value. Without diagnosing sample ratio mismatch, the team might conclude that the treatment hurt revenue.

Hidden segment imbalance

The overall split appears close to 50/50, but slicing by device reveals a strong imbalance among tablet users. If tablets represent a strategically important user segment, perhaps one with higher average order value, this localized SRM still distorts experiment results even though the top level numbers look fine.

Uneven rollout timing

If one variant of the experiment is rolled out earlier than the other, users exposed during the earlier period can cause an imbalance. For example, if the control group is live for several hours before the treatment launches, the total user counts will skew toward control. This timing mismatch can create a sample ratio mismatch that falsely signals an issue with randomization, when in fact it is a deployment sequencing problem.

User ID inconsistencies

Inconsistent or corrupted user identifiers can cause users to be assigned to different groups across sessions or devices. For instance, if a user's ID changes due to cookie resets or login status changes, the experiment platform might count them multiple times or misassign them, leading to an SRM. Ensuring stable and consistent user IDs is critical for accurate group assignment and reliable experiment results.

In each of these cases, naive interpretation of data points without SRM awareness would produce incorrect conclusions about treatment performance.

Best practices to prevent and handle SRM

Standardize randomization logic

Standardize randomization logic in a single service or library. When the assignment code is duplicated across multiple applications, inconsistencies creep in. A centralized assignment stage reduces this risk by ensuring all experiments use the same reliable method for user allocation, minimizing discrepancies that can cause sample ratio mismatch.

Run dry run or shadow tests

Run dry run or shadow tests where variants are hidden but assignment and logging are exercised. This allows you to check that the sample ratio remains correct against historical data and expected counts before real launches. These tests help catch issues early in a controlled environment, preventing SRM from occurring in live experiments.

Routinely verify traffic allocations

Experimenters should routinely verify traffic allocations against designed ratios. Set up monitoring on exposure counts, SRM p-value, and related health metrics with alerts that trigger when p-values cross your chosen threshold. Real-time alerting can help prevent SRM in A/B tests by catching problems before they contaminate weeks of data. DoorDash saw a 70 percent drop in SRM incidents after implementing systematic monitoring, demonstrating the practical value of this approach.

Document root causes and build checklists

Document typical root causes and build a checklist for knowledge management. When SRM appears, engineers and analysts should have a repeatable investigation process rather than starting from scratch each time. This organized approach speeds up diagnosis and resolution, reducing downtime and improving experiment reliability.

Pause or stop experiments with confirmed SRM

Pause or stop experiments with confirmed SRM. Ideally, restart them after the underlying implementation issue is resolved and data collection begins fresh. Continuing experiments with unresolved SRM risks invalid conclusions and wasted resources.

Cautiously apply advanced techniques to salvage insights

While there are advanced techniques to salvage some insights from SRM-affected tests, such as restricting analysis to unaffected segments or applying weighted regression to adjust for the imbalance, these should be used cautiously and clearly labeled as exploratory. They are not substitutes for fixing the root cause, but can provide preliminary guidance when restarting is not immediately possible.

Key metrics for understanding SRM

Here is a concise list of quantitative signals to watch when monitoring for sample ratio mismatch.

Real-time dashboards help monitor these metrics continuously. Tracking them at both the global and segment level, using real-time analytics where possible, lets you detect SRM before it accumulates enough biased data to affect decisions.

Sample ratio mismatch and related concepts

SRM sits within a broader ecosystem of experimentation and statistical concepts. Understanding the connections helps put SRM detection in context.

SRM is often a visible symptom of selection bias. When certain user attributes make it more likely for someone to appear in one variant, the resulting data no longer supports causal claims. The concept parallels survivorship bias: during World War II, analysts at Columbia University studied returning bombers and initially proposed armoring the areas with the most bullet holes. Abraham Wald realized they should focus on the missing planes, the ones that did not return, because estimating plane vulnerability based only on surviving aircraft ignored the most critical damage zones. Similarly, when users go missing from one experiment group, the observed data tells an incomplete story.

SRM p values relate to standard hypothesis testing concepts like confidence level and statistical significance. However, the SRM p value measures assignment balance, not treatment effect. These are distinct questions.

SRM checks complement other diagnostics like power analysis, minimum detectable effect planning, guardrail metrics, and counterfactual logging. Research published through venues like the international conference on knowledge discovery and through Cambridge University Press has explored how regression modeling helps identify imbalances in treatment assignments and how orthogonal effects between overlapping experiments can cause or mask SRM.

SRM detection also connects to canary testing and feature flag rollouts, because the same randomization and logging issues that cause SRM in experiments can affect gradual releases. Think of SRM detection as a mandatory precondition before trusting any effect size or uplift estimate from an A/B test.

Key takeaways

• Sample ratio mismatch occurs when observed group sizes in an experiment deviate from the planned sample ratio more than random variation can reasonably explain.

• SRM is strong evidence of selection bias or an implementation issue in assignment, logging, or analysis, and it undermines experimental integrity.

• The standard way to detect SRM is to run a chi squared test and monitor the resulting SRM p value over time, ideally with automated alerts.

• Preventing SRM through robust randomization, consistent identifiers, careful targeting, and reliable data pipelines saves significant time and avoids misleading experiment results.

• Every mature experimentation program should treat SRM checks as a built in safeguard before drawing conclusions from test data.

FAQs about Sample Ratio Mismatch