[ Statistics, Illustrated ]

The Noise Problem

Most of what makes your metric noisy has nothing to do with your treatment. What if you could just remove it?

A/B tests often take longer than they need to, not because the real effect is small, but because the metric itself is noisy for reasons that have nothing to do with the treatment. This project runs a live simulated experiment twice, once on the raw metric, once adjusted with CUPED, a real, widely-used variance-reduction technique, to measure exactly how much faster and more reliably it finds a genuine effect, without inflating false positives.

Run the experiment See the speed study

Before we start: four terms this page leans on

Each one gets a plain-language sentence first, that's the whole idea if you only want the whole idea, and a more precise technical line underneath it for anyone who wants the exact definition.

Covariate: A number you already know about someone before the experiment starts that tends to predict their outcome, like how much they spent last month predicting how much they'll spend this month.; A pre-treatment variable correlated with the outcome. CUPED uses each visitor's own pre-period value of the same metric as its covariate, so it captures exactly the individual variation the experiment doesn't need to re-explain.
Variance reduction: Making a noisy measurement less noisy, so a real difference is easier to spot against the randomness that was already there.; Shrinking Var(Y) using information that predicts Y but isn't affected by treatment. CUPED's specific reduction is exactly (1 − correlation²), see Methodology.
Statistical power: The chance a test actually detects a real effect, when there truly is one. Low power means real improvements often go unnoticed, not because they weren't real, just because the test wasn't sensitive enough to catch them yet.; P(reject the null | the alternative is true), for a given effect size, sample size, and significance threshold. Reducing variance raises power at the same sample size, or reaches the same power with less data.
Pre-period data: Whatever you already knew about each visitor before the experiment began, like their activity in the weeks before they were randomly assigned to A or B.; Data collected strictly before treatment assignment. This is what makes a covariate safe to use: since treatment can't have influenced something measured before it happened, adjusting for it can't introduce bias, only remove predictable noise.

The rest of this page is about what happens when you use pre-period data to strip predictable noise out of a metric, and how much faster a real effect becomes visible when you do.

Why does removing noise help?

Think of a kitchen scale's tare button. Before you weigh an ingredient, you set the bowl on the scale and press tare, it subtracts the bowl's weight so the reading shows only the ingredient. The bowl's weight was never what you cared about, it was just noise sitting on top of the number you actually wanted. CUPED does the same thing to an experiment metric: it subtracts out the part of a visitor's outcome that was already predictable from what you knew about them before the experiment even started, so what's left reflects the treatment more clearly.

It's a personalized tare, though, not a flat one. A golf handicap is the closer analogy for that part: a handicap doesn't subtract the same number of strokes from everyone, it adjusts each player against their own known baseline, so a beginner's 95 and a scratch golfer's 78 can both be judged as "how much did you improve today," not just "who shot the lower number." CUPED adjusts each visitor against their own pre-period baseline, not a single number for the whole experiment.

The same real difference, easier to see with less noise

Same two arms, same real gap between them. Before adjustment the two distributions are wide enough to overlap heavily, hard to tell apart. After adjustment they're narrower, the same real difference just became much easier to see.

What CUPED actually subtracts

A real (small-sample) example: pre-period value clearly predicts the outcome, that diagonal trend. Fit and subtract it, and what's left has no trend at all, only the part of the outcome pre-period data couldn't have told you, less spread out than the raw outcome was.

Why this doesn't cheat

The adjustment subtracts a constant (each visitor's covariate compared to the overall average covariate), the same rule for every visitor regardless of which arm they're in. Subtracting the same rule from both arms can't manufacture a difference that wasn't there, or hide one that was, it only removes variation the pre-period data already explained. That's why it doesn't inflate false positives (see Key Findings), it just makes the real answer easier to see, faster.

One experiment, watched two ways

A simulated A/B test on a continuous metric (think revenue or engagement per visitor), run twice on the exact same simulated data: once on the raw metric, once adjusted with CUPED using each visitor's own pre-period value. Both always land on the same real answer, watch which one gets there faster.

How much faster, really?

Simulate hundreds of independent experiments at a given correlation and traffic level, and measure two separate things: whether a fixed, pre-planned final look finds a real effect (only meaningful when one exists), and, when it does, how many days it took to first become visible.

Key findings

Short version: subtracting each visitor's own predictable, pre-experiment baseline shrinks a metric's variance by exactly (1 − correlation²), doesn't touch the false-positive rate at all, and lets a real effect become visible in meaningfully fewer days, the benefit scaling directly with how predictive the pre-period data actually is.

The mechanic

Does this actually behave the way the theory says, and does it cheat anywhere?

CUPED doesn't inflate false positives
Raw4.3%
CUPED-adjusted4.85%
Across 2,000 simulated null-true experiments (no real difference between A and B), the raw metric's final-day false-positive rate was 4.3%, the CUPED-adjusted metric's was 4.85%, both close to the textbook 5%. Subtracting the same rule from every visitor, regardless of arm, can't manufacture a difference that wasn't there.
Variance reduction follows one exact formula, not a rule of thumb
Directly from Var(Y_cuped) = Var(Y)·(1 − correlation²): at a pre/post correlation of 0.3, only 9% of variance is removed. At 0.5, 25%. At 0.7, 49%. At 0.85, 72.2%. At 0.9, 81%. The formula isn't approximate, it's exact, confirmed here by direct Monte Carlo simulation, not just asserted (see Methodology).

How much faster, really

The same real effect, the same traffic and time budget, measured with and without the adjustment.

At a strong correlation, the practical difference is dramatic
Raw68.9%
CUPED-adjusted99.5%
At a 0.85 correlation, 30 visitors/day/arm, and a modest genuine lift, checking once at a fixed day 10: the raw metric detects it 68.9% of the time, the CUPED-adjusted metric 99.5% of the time. Same visitors, same treatment, same single pre-planned look, a very different chance of actually seeing the effect that's really there.
The benefit scales with correlation, it isn't a fixed bonus
ρ=0.374.6%
ρ=0.686.8%
ρ=0.8599.5%
Same day-10 fixed-look comparison, varying only the pre/post correlation: at ρ=0.3 CUPED barely helps (69.9% raw vs. 74.6% adjusted, both similar). At ρ=0.6 the gap opens up (69.5% vs. 86.8%). At ρ=0.85 it's dramatic (68.9% vs. 99.5%). A weak covariate is nearly free but nearly useless; a strong one is transformative. There's no such thing as "turning CUPED on", only how predictive the pre-period data actually is.
Given enough time, both get there, CUPED just gets there sooner
Raw median day5%
CUPED median day2%
Extend the window to 30 days at the same 0.85 correlation: both methods eventually detect the real effect in almost every trial (99.0% raw, 100% CUPED). The difference shows up in speed, not in whether the effect is eventually found: the median day of first detection is day 5 for the raw metric, day 2 for the CUPED-adjusted one.

Methodology

This section is the full construction and citations behind everything above. If you just want the intuition and the two analogies, see Why does removing noise help? earlier on this page, each section below also opens with a one-line plain-English summary.

The two engines, run in parallel

Every simulated day, the exact same accumulated data gets tested twice: once as-is, once with each visitor's pre-period value subtracted out.

Each simulated day, every visitor is generated as a correlated (pre-period, post-period) pair, the post-period value is the only one treatment ever touches. A two-sample z-test on means (the same twoSampleZTestOnMeans used by the revenue-metric mode in The Peeking Problem) runs on the raw post-period values, and separately on the CUPED-adjusted values, both reading off the identical underlying simulated visitors, so any difference between the two lines is attributable to the adjustment alone, not to different data.

How theta gets estimated

The one number the adjustment needs, theta, is re-estimated each day from all the data collected so far, pooled across both arms.

θ = Cov(X, Y) / Var(X) is recomputed each day from every visitor observed so far in both arms combined, on the reasoning that the pre/post relationship is a property of the metric itself, not something the treatment is assumed to change. A production implementation more commonly fits θ once, from a genuinely separate historical pre-experiment window, rather than re-estimating it in-experiment as done here, a deliberate simplification noted again in Limitations.

Why this doesn't inflate false positives

The adjustment subtracts the same rule from every visitor regardless of arm, so it can't manufacture or hide a real difference.

CUPED's adjustment, Y − θ(X − mean(X)), applies the identical rule to every visitor in both arms. Subtracting a quantity that doesn't depend on which arm a visitor was assigned to shifts both arms' means by the same expected amount (zero, in expectation), it cannot introduce a systematic difference between arms that wasn't already there. What it removes is variance correlated with X, which is exactly why the false-positive rate stays at its nominal level (see Key Findings) while the effect becomes easier to detect.

The repeated-trials speed study measures two different things on purpose

"Did a fixed final look detect it" and "how many days did it take to first cross significance" are kept separate, one is safe to interpret as a false-positive rate, the other isn't.

This page reports a final-day-only detection rate (a single, pre-committed look, exactly like The Peeking Problem's "checked once" bar) for the false-positive-rate claim, and, separately, a first-crossing day for the speed story. Conflating the two would silently reintroduce The Peeking Problem's own peeking inflation into a page that isn't about peeking at all, first-crossing search is itself a form of continuous monitoring. That distinction wasn't obvious until this page's own test suite caught it: an early version of the repeated-trials engine used first-crossing to compute the null false-positive rate too, and it came back at 25%, not 5%, exactly the kind of inflation the sibling project describes. The fix keeps the two questions, and the two decision rules behind them, separate.

What the published research says

This is a real, widely-used technique, not a novel construction, first published by Microsoft's experimentation team.

CUPED (Controlled-experiment Using Pre-Experiment Data) was introduced by Deng, Xu, Kohavi, and Walker in "Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data" (WSDM 2013), and is in production use across major online experimentation platforms. The variance-reduction identity and the unbiasedness argument used throughout this page follow that paper's construction directly.

Limitations

This simulates a single continuous metric with a genuinely linear pre/post relationship at a known, fixed correlation. Real metrics can have a non-linear or heteroskedastic relationship to their pre-period value, where a linear CUPED adjustment captures less of the available variance reduction than the clean correlation number here implies.
θ is re-estimated in-experiment, pooled across both arms, each simulated day, rather than fit once from a genuinely separate historical pre-experiment window, the more common production pattern. Re-estimating in-experiment is a reasonable simplification for this demo, not a difference that should matter much in practice, but it's a real design choice, not the only correct one.
The core mechanic requires a real pre-treatment covariate to exist for every visitor. New visitors with no pre-period history (a very common case in practice) get little or no benefit from this specific technique; other variance-reduction methods exist for exactly that situation and aren't simulated here.
This engine generates each simulated visitor individually rather than using the aggregate-sampling shortcuts The Peeking Problem's revenue mode needed for 500,000 visitors/day. Traffic here is capped in the low hundreds per day, plenty to see the real effect, not optimized for that scale.

Reflection & next steps

What was surprising: How directly The Peeking Problem's own lesson reappeared here, uninvited. An early version of the repeated-trials study used a first-crossing search to compute the null false-positive rate, and it came back at 25%, not 5%. Continuous monitoring inflates false positives whether or not the page in front of you is about peeking.
Hardest part to get right: Keeping "did a fixed final look detect it" and "how many days until first crossing" as two genuinely separate measurements, computed from the same simulated data but never blended into one number. It would have been easy to report a single, more impressive-sounding "days to detect" statistic that quietly smuggled in an adaptive stopping rule.
What a proper fix would take: θ is estimated from in-experiment, pooled data here rather than a genuinely separate historical pre-period, and the covariate is assumed linear and always available. A natural follow-up would fit θ from real historical data before the experiment starts, and explore what happens to the benefit when a meaningful fraction of visitors have no pre-period history at all.