[ Statistics, Illustrated ]

The Peeking Problem

What happens when you check your A/B test every day, and can anything actually fix it?

Checking results early feels harmless: you're just looking. This project runs a live simulated A/B test watched by three statistical engines at once, frequentist, Bayesian, and a third built specifically to stay honest under continuous monitoring, then repeats the experiment hundreds of times to measure, with real numbers, what continuous peeking actually costs under each one, and whether the fix really works.

Run the experiment See the peeking study

Before we start: five terms this page leans on

Everything below uses these five words a lot. Each one gets a plain-language sentence first, that's the whole idea if you only want the whole idea, and a more precise technical line underneath it for anyone who wants the exact definition.

False positive: Concluding B is actually better than A when, in truth, they're exactly the same, you just happened to see unlucky data.; Rejecting the null hypothesis, or crossing a Bayesian decision threshold, when the null is actually true. The nominal rate tested against throughout this page is 5%.
p-value (the frequentist number): A roundabout question: if A and B were truly identical, how surprising would data like this be? Small means surprising, treated as evidence something's really different. It is not the probability that B is actually better, that's a common misreading.; P(data at least this extreme | the null hypothesis is true), computed here from a two-proportion or two-sample z-test recomputed on the cumulative data each simulated day.
Posterior probability (the Bayesian number): A direct probability that B is actually better than A, given the data seen so far and a starting assumption about how likely that was before looking. This is the number people actually want when they ask a p-value's question.; P(rate_B > rate_A | data, prior), estimated here by Monte Carlo sampling from each arm's posterior distribution (Beta-Binomial, or Beta × Normal-Normal for a revenue metric).
Bayes factor: A measure of how much the data moved your belief, not just where your belief ended up: the ratio of how likely the data is under "something changed" versus under "nothing changed." That distinction is what the rest of this page turns on.; The likelihood ratio of the observed data under a mixture of alternative effect sizes versus under the null. Unlike a raw posterior probability, it's constructed to be a martingale under the null, see "Can you actually peek safely?" below.
Peeking: Checking your results early and often, and being willing to act the moment they look convincing, instead of waiting for one look at a pre-planned end date.; Optional stopping: a decision rule whose stopping time depends on the data itself rather than being fixed in advance, which invalidates the usual guarantees behind a single-look p-value or a naive posterior threshold.

The rest of this page is about what checking early does to each of these numbers, and whether there's a way to check early safely.

Can you actually peek safely?

Yes, but only with the right kind of test. The two sketches below are the whole mechanism, in miniature: why watching a normal test every day is dangerous, and what a genuinely different construction does instead.

A wandering statistic will eventually cross any line

This is pure noise, no real difference between A and B, checked every day. A p-value or a raw posterior probability wanders like this under the null. Wait long enough, and it is guaranteed to cross any fixed line eventually, purely by chance. That's the actual mechanism behind "checking every day inflates false positives."

A fair game stays bounded, almost always

These are many possible runs of a test built as a "fair game": engineered so its expected value under the null never drifts upward, however often it's checked. Statisticians call this a martingale, a process where, however things have gone so far, your best guess for what happens next is more of the same, no drift up or down. That specific property, not "being Bayesian," is what Ville's inequality turns into a real, provable bound on how often it ever crosses a high bar.

What a Bayes factor mixture actually weighs

Instead of testing one guess about the effect size, the always-valid test blends the likelihood of the data across a whole range of plausible effects, weighted by how plausible each one is. That blending, not the Bayesian-ness by itself, is what makes the resulting ratio behave like the fair game above instead of the wandering statistic.

So, can you peek safely?

Yes, if: you're using a genuine mixture Bayes factor, the fair-game construction above, this page's third engine.
No, if: you're watching a raw posterior probability cross a fixed threshold. That's the same wandering-statistic problem as a p-value, wearing different clothes (see Key Findings).
The guarantee breaks, not just weakens, if: the underlying model is badly misspecified, or there's an unprincipled "I'll act early if it's extreme enough" override sitting outside the formal rule.
It costs you, without breaking, if: the assumed effect size (τ) is badly mismatched to reality. Safety holds regardless; you pay for the mismatch in how quickly you can detect a real effect (see Key Findings).

What makes a prior trustworthy?

Picture a doctor seeing a patient before running any test. They already have a rough sense of how likely different diagnoses are, based on the patient's age, symptoms, and what's common in general, formed before any test result comes back. That's a prior. The test result then updates it.

On this page, the prior is a starting guess about each page version's conversion rate (or revenue), before any of this experiment's visitors have shown up. The prior-strength slider in the panels below controls how much that starting guess resists being moved by new data, the same way a doctor who's seen a thousand similar cases moves less than one who's guessing.

Before you trust a prior, ask

Is it based on something real, like this page's own history, not a guess or a hope?
Is its confidence level honest? If you're not actually sure, the prior shouldn't act sure.
Would you be comfortable if this prior alone, with zero new data, made the decision?

Try it yourself: drag prior strength, what the data alone suggests, and how many visitors have been observed, and watch where the resulting posterior actually lands.

Prior strength: weak → strongWhat the data alone suggestsVisitors observed so far

Assumed baseline: 10%, matching the default above. The prior is always centered there; only its concentration changes as you drag its strength.

A weak prior barely resists: the posterior lands close to whatever the data alone suggests, fast to react, including fast to overreact to early noise, which is exactly why a weak Bayesian prior offers so little protection against peeking (see Key Findings). A strong prior resists more: the posterior stays closer to the assumed baseline until real evidence accumulates, which cuts the any-day false-positive rate from 34.6% down to 17.0% at this page's default scale, at a real cost, correctly detecting a genuine 20% lift drops from 42.0% to 36.3% by day 30. Drag "visitors observed" up and even a strong prior gets overwhelmed: at real production traffic volumes, a prior sized to matter at modest traffic becomes functionally weightless within the first day or two (see Methodology).

None of that says whether the prior itself was right. Picture that same doctor, but their experience actually comes from a small, biased, or outdated set of patients, they still walk in confident, and that confidence now works against the patient: the more strongly they hold a wrong assumption, the more real evidence it takes to correct them. Same here: if a prior's central guess is wrong, or its source data was contaminated (a buggy prior experiment, a different audience, a seasonal fluke), a strong prior doesn't just resist noise, it resists the correction too. A weak, wrong prior is comparatively harmless, it barely resists anything, real data takes over quickly. Strong and wrong is the dangerous combination, not strong by itself, confidently wrong beats honestly uncertain in exactly the wrong direction.

That's the exact tug-of-war the prior-strength slider in the panels below controls.

One experiment, watched three ways

A simulated A/B test, on a revenue-per-user metric by default (most visitors spend $0, the rest are right-skewed) or a simple conversion rate, with three engines re-checking the same accumulating data every simulated day: a frequentist p-value, a Bayesian probability that B beats A, and an always-valid p-value built to stay honest under continuous monitoring. Change the settings or press run to generate a new random experiment. Watch the always-valid line especially, it's the fair-game construction from Can you actually peek safely? above, running on real numbers.

What peeking actually costs you, and doesn't

Simulate hundreds of independent experiments where there is truly no difference between A and B, and count how often each of the three methods would have declared a winner: checking once at a fixed end date, versus checking every single day and stopping the moment any of them looked convincing. This is the wandering-statistic-vs-fair-game story from Can you actually peek safely?, run hundreds of times over.

Very high visitor counts (100,000+/day) take a few seconds to compute in the browser: each run simulates hundreds of full 30-day experiments.

Key findings

Short version: a fixed Bayesian threshold barely protects you, checking every day inflates it almost as much as a plain p-value does, but a genuine Bayes factor, the always-valid test, actually holds the line, at a real cost in how fast it can detect a genuine effect.

The problem

Checking a normal A/B test every day inflates how often it cries wolf, whether you're using a p-value or a plain Bayesian probability.

Checking every day is almost as risky with a fixed Bayesian threshold as with a plain frequentist test
Frequentist, once5.1%
Frequentist, daily27.2%
Bayesian, once5.5%
Bayesian, daily34.6%
At default settings (10% baseline rate, 50 visitors/day per arm, 30 simulated days), checking once at a fixed end date lands close to the textbook 5% false-positive rate for both methods (5.1% frequentist, 5.5% Bayesian with a weak prior). Checking every day and stopping at the first convincing result inflates both to roughly 27–35%, a 5–6x increase either way. A Bayesian probability crossing a fixed threshold offers essentially no built-in protection against peeking on its own.
False positives cluster early
At the default scale, the median day of a first false positive is day 5 of 30 for both methods, and 69% of all frequentist false positives happen within the first third of the experiment. The moment it feels most tempting to peek, early in a test, is also the riskiest moment to trust what you see.
This matches independently published numbers
Evan Miller's widely cited "How Not To Run An A/B Test" reports roughly a 26% false-positive rate when checking after every single observation at a nominal 5% threshold. This simulation's own model, built independently, lands at a comparable 27–29% for daily checks over 30 days. A 2025 simulation by Alex Molas found a fixed Bayesian posterior threshold similarly overshoots its nominal rate under peeking, consistent with the pattern shown here.

The fix, and why it actually holds

A genuinely different kind of test, not just a smaller version of the same problem.

A properly constructed Bayes factor actually fixes it
Always-valid1.3%
Frequentist27.2%
Fixed Bayesian34.6%
Add a third method, an "always-valid" test built from a genuine Bayes factor (a mixture likelihood ratio over a prior on the effect size, not a posterior-probability threshold), and the story changes completely: at the same default settings, its any-day rate is 1.3%, next to 27.2% for the frequentist test and 34.6% for the fixed Bayesian threshold. This isn't a smaller version of the same problem, it's a genuinely different kind of guarantee, provable rather than merely observed to be smaller (see Methodology).
That guarantee holds up even if you keep watching for months
Frequentist, 120d39.4%
Fixed Bayesian, 120d49.3%
Always-valid, 120d2.55%
Extend the monitoring window from 20 to 120 days and the naive methods get worse the longer you watch: the frequentist any-day rate climbs from 25.4% to 39.4%, the fixed Bayesian threshold from 32.6% to 49.3%. The always-valid test's any-day rate also rises, but from 0.9% to 2.55%, staying comfortably under the 5% nominal rate the entire time. More looks make the naive methods meaningfully worse; they barely move this one.

What the fix costs

Safety isn't free. Here's exactly what it trades away, and when.

The fix has a real cost: it's slower to detect a real difference, not a free lunch
Freq./Bayesian, day 3041%
Always-valid, day 3010.3%
Against a genuine 20% relative lift, by day 30 the frequentist and fixed Bayesian tests each detect it in about 41% of trials; the always-valid test detects it in 10.3%. That gap narrows with more data (66% vs. 94% by day 120) but doesn't close. A test built to stay safe under unlimited peeking is, at any fixed sample size, less eager to declare a winner than a test that isn't safe. That's the actual trade, not a defect to route around.
A skeptical prior softens the problem, but the real cost shows up only when there's a genuine effect to detect
Raising the Bayesian prior's strength cuts the any-day false-positive rate from 34.6% (weak) to 24.0% (mild) to 17.0% (strong), while the final-day-only rate stays close to the nominal 5% throughout (5.5%, 4.9%, 4.7%). So the prior isn't just quietly failing to detect real effects under the null, that would show up as a falling final-day rate, and it doesn't. Run the same comparison under a genuine 20% lift instead, and the cost appears exactly where it should: a weak prior correctly detects the real effect by day 30 in 42.0% of trials, dropping to 38.4% (mild) and 36.3% (strong prior). A stronger prior trades detection power for peeking resistance, not single-look accuracy.
That protection depends entirely on how the prior's weight compares to your daily traffic
The same “strong” prior that cuts the any-day rate from 34.6% to 17.0% at 50 visitors/day per arm does essentially nothing at 5,000/day (32.9% vs. 32.5%) and literally nothing at 500,000/day (33.5% vs. 33.5%, identical to three significant figures). A prior is a fixed number of pseudo-observations; once a single day of real traffic outweighs it by orders of magnitude, it gets overwhelmed within the first day or two and stops influencing anything. Try it yourself in the peeking study above: a prior sized to matter at modest traffic is functionally weightless at real production scale.

Does this generalize?

Same story, tested against a messier metric, more traffic, and a much longer time window.

The same story holds for revenue per user, not just a clean conversion rate
Frequentist, daily26.6%
Fixed Bayesian, daily32.7%
Always-valid, daily0.47%
Switch the metric to revenue per user (the default above: 1% of visitors spend anything at all, the rest right-skewed around a $50 average), at a realistic 500,000 visitors/day, and the pattern repeats almost exactly. Frequentist: 4.6% checking once, 26.6% checking every day (5.8x). Fixed Bayesian threshold: 5.5% to 32.7% (5.9x). Always-valid: 0.07% to 0.47%, staying in the same dramatically-lower regime it holds for a simple conversion rate. Zero-inflation and heavy skew change what the metric looks like, not whether checking every day inflates a threshold-based test, or whether a genuine Bayes factor resists it.
The core false-positive inflation barely moves across baseline rate or traffic volume
A 10% baseline at 50 visitors/day, a 1% baseline at 5,000/day, and a 1% baseline at 500,000/day all produced the same story for the frequentist test: a final-day-only rate near the nominal 5% (4.9–5.2%), inflating to 27–28% under daily peeking, a consistent 5.3–5.5x. The classical asymptotic behavior behind this holds even at the sparsest scale tested here (about 5 expected successes per arm per day).

Methodology

This section is the full derivation, exact theorem, and citations behind everything above. If you just want the intuition and the direct answer, see Can you actually peek safely? earlier on this page, each section below also opens with a one-line plain-English summary.

The first two engines

The frequentist and naive-Bayesian numbers on this page both come from recomputing a standard test on the same growing pile of data, once per simulated day.

For a conversion-rate metric, each simulated day a pooled two-proportion z-test recomputes a two-sided p-value on the cumulative data, and a Beta-Binomial model updates each arm's posterior conversion-rate distribution, from which P(B beats A) is estimated by Monte Carlo sampling (1,500 posterior draws per day in the live panel, 200 in the repeated-trials study to stay responsive). Both engines see exactly the same accumulating data at exactly the same time. A third engine, described below, watches too.

Modeling a revenue-style metric

Revenue per user gets modeled as two questions stacked together: did they spend anything, and if so, how much.

Revenue per user isn't a proportion, most visitors contribute exactly $0, and the rest are right-skewed. This project models it as a standard two-part ("hurdle") model: whether a visitor spends anything at all is the same Beta-Binomial machinery as the conversion-rate engine, and how much they spend given they do is modeled as log-normal (a fixed, shared spread across arms, a simplification: most treatments change whether or how much people spend, not the underlying shape of spend variability). The frequentist test becomes a two-sample z-test on mean revenue per user; the Bayesian side adds a second, jointly-sampled posterior for the log-revenue mean; the always-valid test needs no new theory at all, it already operates on a raw difference and variance, whichever metric they came from.

Why a 0.975 Bayesian threshold, not 0.95

0.95 quietly gives the Bayesian rule an easier bar to clear than the frequentist test's 5%, so it's recalibrated to 0.975 to keep the comparison fair.

A symmetric two-sided decision rule (declare a winner when P(B beats A) is above the threshold or below one minus it) needs its threshold calibrated before the two methods can be compared fairly. At 0.95 the Bayesian rule's single-look false-positive rate runs closer to 10%, roughly double the frequentist test's 5%, since two one-sided 95% checks compound. Calibrating to 0.975 instead brings the Bayesian single-look rate back in line with the frequentist test's, so the "checking every day vs. checking once" comparison on this page is apples to apples rather than comparing two differently calibrated tests.

What makes a prior "weak" or "strong" here

The prior slider controls how much real evidence it takes to move either arm's estimate away from the assumed baseline (see "What makes a prior trustworthy?" above for the intuition and a playground to try it).

A common, principled way to encode "skeptical that A and B differ" is a prior directly on the lift (B minus A), centered at zero: weak means wide and diffuse, strong means tightly concentrated at zero. This project does something related but not identical: both arms start from a Beta prior whose mean equals the assumed baseline rate (not 50%, and not the lift), so a strong prior doesn't fight the true baseline the way a prior fixed at 50% would at, say, a 1% conversion rate. What the slider controls is the prior's concentration (a pseudo-observation count, from 1 at the weak end to 200 at the strong end): weak takes almost no real data to move away from the baseline; strong takes substantially more evidence before either arm's estimate is allowed to drift. It's the same spirit as a lift-centered prior, skepticism that either arm has really changed, expressed as skepticism that either arm has moved from a shared starting point, rather than as a prior on the difference itself.

Why the prior's effect depends on your traffic volume

A prior is a fixed amount of pretend data, so the more real traffic a test gets, the less that fixed amount matters.

A pseudo-observation count is a fixed quantity (at most 200 here), and Bayesian updating simply weighs it against however much real data has accumulated. At 50 visitors/day, 200 pseudo-observations are comparable to several days of real traffic, so the prior meaningfully shapes the posterior for a while. At 5,000 or 500,000 visitors/day, 200 pseudo-observations are dwarfed by a single day's real data, so the prior's influence disappears almost immediately regardless of how "strong" it's set. This isn't a quirk of this particular simulation, it's a direct, general consequence of what a fixed-size prior actually is: a prior sized to matter at one traffic volume needs to be resized to matter at another.

The repeated-trials peeking study

The centerpiece chart runs hundreds of experiments where A and B are secretly identical, then checks how often each method got fooled anyway.

The centerpiece comparison runs several hundred independent simulated experiments where the two arms genuinely share the same true rate (the null is true by construction), then measures how often each method would have called a winner: checking once at the end, versus checking every day and stopping at the first convincing result. Baseline rate, daily traffic, and prior strength are all adjustable above, so the comparison can be run at whatever scale is relevant, not just the default one. This measures the classic "optional stopping" problem directly instead of asserting it.

Peeking isn't the problem, acting on it early is

Looking at a dashboard isn't what breaks a test, deciding to act on what you see before the planned end date is.

The two bars in the chart above aren't "looked at the data" vs. "didn't", interim results exist every day in both scenarios. The "checked once (end)" bar is what happens when the decision rule ignores those interim results and only depends on the final day: it sits right at the nominal rate, because the decision was fixed in advance and never actually reacted to anything that happened before day 30. The "checked every day" bar is what happens when the decision rule reacts to whichever day looks most convincing. So watching a dashboard every day out of curiosity, while genuinely waiting for a pre-committed end date to decide, isn't the problem this page describes. That guarantee has zero tolerance for exceptions, though: even a rare "I'd stop early if it looked extreme enough" escape hatch reintroduces the same inflation, in proportion to how likely that escape hatch actually is to fire.

A third engine: an always-valid test

The always-valid test is built from the ground up to be the fair-game construction shown earlier, not just a smaller version of the other two.

At each day, this engine computes a genuine Bayes factor exactly as pictured in "Can you actually peek safely?" above: the likelihood of the observed difference under a Normal(0, τ²) mixture over possible true effect sizes, divided by the likelihood under "no effect." τ, how large an effect this test is tuned to detect efficiently, is derived automatically here from the baseline rate (tuned around a ~30% relative lift) rather than exposed as its own control; a real analysis would choose it deliberately.

Why this one is provably different

This is the theorem behind the fair-game diagram above: a real, provable cap on how often it's ever fooled, not just an empirical pattern that happened to hold here.

This mixture likelihood ratio is exactly the martingale pictured above: a nonnegative process whose expected value stays exactly 1 at every fixed point in time under the null, however the data turns out. Ville's inequality turns that property into an exact theorem: the probability this ratio ever crosses 1/α, checked at any time, however often, is bounded by α, for every possible stopping rule at once, not an empirical pattern that happens to hold in this particular simulation. The numbers in Key Findings confirm the theorem actually plays out in a real repeated-trials simulation, they don't substitute for it.

What the published literature says, and what actually resolves the Rouder question

Two seemingly contradictory academic claims about Bayes and peeking are both right, they were just talking about two different kinds of Bayesian test.

Optional stopping inflating frequentist false-positive rates is a well-established result dating to Armitage, McPherson and Rowe (1969), popularized for practitioners by Evan Miller's "How Not To Run An A/B Test" (2010) and formalized into "always-valid" sequential testing by Johari, Pekelis and Walsh (2017), the exact construction used above and the basis for Optimizely's Stats Engine. Rouder (2014) argued Bayes factors resist optional stopping, while de Heide and Grünwald (2021) and Sanborn and Hills (2014) showed a fixed posterior-probability threshold doesn't get that protection for free. Both are right, about different things: Rouder's claim is specifically about genuine Bayes factors like the mixture likelihood ratio built here, and this page's own numbers show exactly that distinction in practice, a raw P(B beats A) threshold inflates under peeking, a real Bayes factor over a mixture prior on the effect doesn't. Deng, Lu and Chen (2016, Microsoft) make the same distinction, explicitly calling a fixed posterior-threshold stopping rule a "bad practice" separate from a properly designed sequential Bayesian test.

Limitations

This simulates one metric (a single conversion rate) with a stationary true rate across the whole run. Real experiments often track several metrics at once and can drift over time. Testing many metrics simultaneously compounds a different, related problem, false discovery across metrics, which isn't simulated here.
The Bayesian posterior probability at each day is a Monte Carlo estimate (1,500 samples in the live panel, 250 in the repeated-trials study to stay responsive), not a closed-form number. It has its own sampling noise, smaller than the effects demonstrated here, but real; see the stats-sim test suite for a direct check of how that noise shrinks as sample count grows.
The always-valid test's mixing variance (τ², how large an effect it's tuned to detect efficiently) is auto-derived from the baseline rate here rather than exposed as its own control. A real analysis should choose it deliberately, tuned to the smallest effect actually worth detecting, since a mismatched τ costs real power (see Key Findings). Other valid constructions exist too, group sequential designs with alpha-spending functions, and sequential Bayes factor designs with different mixing choices, that this page doesn't build out; the one implemented here is a real, representative example, not the only correct construction.
Some vendors describe their Bayesian A/B testing tools as immune to peeking outright. That claim can be true of an actual engineered product, if it implements a proper sequential correction internally, while being false of the fixed posterior-threshold rule demonstrated on this page. Worth checking what's actually running underneath before taking a vendor's calibration claims at face value.

Reflection & next steps

What was surprising: How similar the frequentist and Bayesian inflation numbers were under a weak prior, given how often "Bayesian solves peeking" gets repeated as received wisdom. The published literature actually settled this more carefully than the popular version suggests (see Methodology).
Hardest part to get right: Calibrating the Bayesian decision threshold to a comparable single-look false-positive rate before trusting any "any day vs. final day" comparison between the two methods. It would have been easy to compare an under-calibrated 0.95 threshold against a well-calibrated frequentist test and draw the wrong conclusion.
What a proper fix would take: The always-valid test here uses one fixed, auto-derived choice of mixing variance, tuned generically from the baseline rate rather than a deliberately chosen minimum-effect-of-interest. A natural follow-up would expose that as a real control and show how the power/safety trade shifts as it's tuned differently, alongside a genuinely different kind of fix: a decision-theoretic, expected-loss-based stopping rule that weighs the size of being wrong, not just the probability of it.