[ Statistics, Illustrated ]

The Dashboard Problem

Test twenty metrics at once, and something will look like a winner by chance, even when nothing real changed.

Checking one metric at a fixed 5% false-positive rate feels safe. Checking twenty at once, even at that same safe rate each, isn't, the chance that at least one of them cries wolf climbs fast. This project simulates a real dashboard of metrics, some genuinely moved, most genuinely not, and measures exactly how often naive thresholds get fooled, and how much two real corrections, Bonferroni and Benjamini-Hochberg, actually fix it.

Run the experiment See the metric-count study

Before we start: three terms this page leans on

Each one gets a plain-language sentence first, that's the whole idea if you only want the whole idea, and a more precise technical line underneath it for anyone who wants the exact definition.

Family-wise error rate (FWER): The chance that at least one metric out of a whole batch looks like a winner purely by chance, even though nothing actually changed anywhere.; P(at least one false rejection among m simultaneous tests). Grows toward 1 − (1 − α)^m as m increases, even though each individual test still runs at the same α.
False discovery rate (FDR): Instead of asking "did anything go wrong at all," this asks "of the metrics I'm calling winners, what fraction are actually mistakes?" A more forgiving, often more useful question once you're testing a lot of metrics at once.; E[false rejections / total rejections], with the ratio defined as 0 when nothing was rejected. Controlling FDR at α means: on average, no more than an α fraction of whatever gets flagged as significant is actually a false positive.
False positive, for a batch of metrics: The same idea as a single false positive, a metric that looks like it moved when it actually didn't, just now happening somewhere inside a whole dashboard of metrics checked at once, not just one.; Rejecting the null for a metric where the null is actually true. What's different here isn't the definition, it's that it's now one event among many simultaneous tests, which is exactly what makes the at-least-one version (FWER) and the proportion version (FDR) two different, both useful, ways to measure it.

The rest of this page is about what happens to these two numbers as you test more metrics at once, and what two real corrections actually do about it.

Why does testing more metrics find more "winners"?

The Texas sharpshooter fallacy is named for a (probably apocryphal) marksman who fires a bunch of shots at the side of a barn, finds wherever a few of them happened to cluster, then paints a bullseye around that cluster afterward and calls himself a sharpshooter. Fire enough shots, and some cluster is guaranteed to show up somewhere, that's not skill, it's arithmetic. Testing twenty metrics at once and reporting whichever one came back significant is the same move: the "bullseye" gets painted after you already know where the arrows landed.

Here's a way to make that concrete: ask 1,000 people to each call ten coin flips in a row, out loud, in advance. By pure chance alone, about one of them will get all ten right. That person isn't psychic, you just asked enough people. Testing enough metrics at once has exactly the same arithmetic hiding inside it.

Fire enough shots, something clusters

All 18 shots were fired the same way, at random. Three of them happened to land near each other, purely by chance. The bullseye only exists because it was drawn around that cluster afterward, it wasn't aimed at.

1,000 people call ten coin flips each

A real binomial(10, 0.5) distribution, not a stylized sketch: most people land near 5 of 10, and getting all 10 right happens for about 1 in 1,024 people, purely from guessing. Ask enough people (or test enough metrics), and that rare tail stops being rare in absolute terms, it becomes a near-certainty that someone, or something, lands there.

The fix isn't "test fewer metrics"

The honest fix isn't to stop measuring things, it's to adjust the bar for calling something a winner based on how many things you're checking at once. Bonferroni does that bluntly (divide the threshold by the number of tests); Benjamini-Hochberg does it more cleverly (control the proportion of winners that are likely fake, rather than the chance of any fake winner at all). Both are simulated for real below, not asserted.

One dashboard, three decision rules

A simulated dashboard of N metrics from a single experiment, most genuinely unchanged, a few genuinely lifted, each with its own real p-value from real simulated data. Watch which ones each of the three rules, no correction, Bonferroni, and Benjamini-Hochberg, would flag as a winner.

What happens as you test more metrics

Simulate hundreds of independent dashboards at each of several metric counts (2, 5, 10, 20, 30, 50), and measure how often each rule gets fooled by at least one false discovery, and how many of the genuinely real effects it still manages to find.

Each run simulates 300 independent dashboards at each of six metric counts, 1,800 simulated experiments total.

Key findings

Short version: checking more metrics at once inflates the chance that at least one looks like a winner by chance, from under 5% at just a couple of metrics to over 80% at fifty. Bonferroni fixes that at a real cost in power; Benjamini-Hochberg controls a different, usually more useful quantity instead, and keeps meaningfully more power as the metric count grows.

The problem

What checking a whole dashboard of metrics, uncorrected, actually costs you.

Testing more metrics finds more "winners", many of them fake
2 metrics4.8%
10 metrics30.8%
20 metrics50.3%
50 metrics83.5%
With 70% of metrics genuinely unchanged and 30% genuinely lifted, the naive rule's chance of flagging at least one false winner climbs from 4.8% at 2 metrics, to 30.8% at 10, 50.3% at 20, and 83.5% at 50. At real dashboard scale, a false discovery isn't an edge case, it's close to guaranteed.
Around one in eight flagged "winners" is fake, at any scale
The naive rule's false discovery rate, the fraction of everything it flags that's actually a false positive, sits between 11.9% and 13.5% regardless of whether 2, 10, or 50 metrics are being tested. More metrics doesn't make each individual flagged result any less trustworthy, it just makes it far more likely that at least one shows up.

The two real fixes, and their trade-off

Bonferroni and Benjamini-Hochberg both work, they just promise different things.

Bonferroni keeps false discoveries rare, at a real cost in power
Naive power71.6%
Bonferroni power21.8%
At 50 metrics, Bonferroni's chance of any false discovery stays at 3.4% (vs. naive's 83.5%), but its power to actually detect the genuinely real effects drops to 21.8%, compared to naive's 71.6%. The stricter per-test threshold that buys safety is the same threshold that buys away sensitivity.
Benjamini-Hochberg keeps roughly double Bonferroni's power at the same scale
Bonferroni power21.8%
BH power43.9%
At 50 metrics, both corrections keep their false discovery rate well under 5% (Bonferroni 1.0%, BH 3.5%), but BH still detects 43.9% of the genuinely real effects, compared to Bonferroni's 21.8%. Controlling the rate of false discoveries, rather than the chance of any at all, buys back real sensitivity.
BH isn't magic: it allows individual false discoveries, it just keeps their proportion capped
BH's own chance of any false discovery (its family-wise error rate) actually climbs with the metric count, from 4.2% at 2 metrics to 23.9% at 50, unlike Bonferroni's, which stays flat. That's the real trade, not a flaw: BH promises something different, the expected proportion of flagged winners that are fake stays capped near the nominal rate (3.4% at 50 metrics), not that a false discovery can never happen at all.

Methodology

This section is the full construction and citations behind everything above. If you just want the intuition and the two analogies, see Why does testing more metrics find more winners? earlier on this page, each section below also opens with a one-line plain-English summary.

Three rules, run on the same simulated dashboard

Every simulated metric gets one real p-value from real simulated data, then all three rules are applied to the exact same batch of p-values.

Each simulated "experiment" generates N metrics, a configurable fraction genuinely null and the rest genuinely lifted, each as its own two-arm binomial comparison (reusing the same binomial sampling and two-proportion z-test used throughout this series), producing one real p-value per metric. Naive rejects whichever p-values individually fall under alpha. Bonferroni rejects whichever fall under alpha divided by the number of metrics. Benjamini-Hochberg sorts the p-values and rejects every one at or below the largest rank whose p-value clears that rank's own line, (rank/N)·alpha, which is why it can reject some p-values individually above alpha/N that Bonferroni would have missed.

Why Bonferroni is conservative

Bonferroni's per-test threshold guarantees the chance of any false discovery stays at alpha, by union-bound logic that doesn't need the tests to be independent.

If each of N true-null tests is run at alpha/N, the probability any one of them falsely rejects is at most N·(alpha/N) = alpha, by the union bound, a simple inequality that holds regardless of whether the tests are independent or correlated. That generality is exactly why it's conservative: it's designed to be safe under the worst case, not tuned to the specific dependence structure of any one dashboard.

What Benjamini-Hochberg actually promises

BH controls the expected proportion of flagged winners that are fake, not the chance of any fake winner existing at all, a different and usually more useful guarantee.

Benjamini and Hochberg's 1995 procedure guarantees E[false rejections / total rejections] <= (m0/m)·alpha under independence (or a common form of positive dependence), where m0 is the true number of null hypotheses, at most alpha regardless. It says nothing about whether any individual false discovery happens, only about their expected share of whatever gets flagged, which is exactly the quantity this page's Key Findings confirm stays controlled even as the metric count climbs.

The repeated-trials metric-count study

The same batch simulation runs hundreds of times at each of six metric counts, and the false-discovery and power numbers are tallied at every count, not asserted from one run.

At each metric count in {2, 5, 10, 20, 30, 50}, 300 independent simulated dashboards are generated (1,800 total per full study run), and for each of the three rules: whether any true-null metric was falsely flagged (feeding the family-wise error rate), and what fraction of genuinely-lifted metrics were correctly flagged (feeding power). Baseline rate, the size of the real lift, and the fraction of genuinely-lifted metrics are all adjustable above.

What the published research says

Benjamini-Hochberg is the actual, widely-cited 1995 paper that introduced controlling the false discovery rate as an alternative to the older, more conservative family-wise corrections.

Benjamini, Y. and Hochberg, Y. (1995), "Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing," Journal of the Royal Statistical Society, Series B, is the paper this page's second correction is built from directly. Bonferroni's correction predates it by decades and remains the standard conservative baseline it's usually compared against, exactly the comparison made throughout this page.

Limitations

Metrics are simulated independently of each other. Real dashboard metrics are often correlated, revenue and conversion rate tend to move together, which changes the exact false-discovery and power numbers (BH's guarantee in particular is proven under independence or a specific form of positive dependence, not arbitrary correlation). The qualitative story, naive inflates, corrections control it at a cost, is robust to this, the precise numbers here are not.
The fraction of metrics that are genuinely lifted is a known input to this simulation, so its ground-truth column can be shown at all. A real dashboard never gets to see that column, that's the actual problem this page is about, not a simplification that changes any of the numbers.
This uses the original 1995 Benjamini-Hochberg procedure. Variants exist for arbitrary dependence structures (Benjamini-Yekutieli) and other refinements: this page implements one real, representative correction, not the only correct one.

Reflection & next steps

What was surprising: That Benjamini-Hochberg's own chance of any false discovery climbs with the metric count, the same failure mode as the naive rule, just slower. It's easy to assume BH "fixes" multiple testing the way Bonferroni does; it actually trades one guarantee for a different, usually more useful one, not a strictly stronger version of the same guarantee.
Hardest part to get right: Getting the false discovery rate's own definition right in code: it's an average over trials where trials with zero rejections contribute exactly 0, not an average only over trials that had rejections, a subtle difference that changes the number. Confirming the observed family-wise error rate against the textbook 1-(1-alpha)^m formula also required remembering that formula only applies when every tested metric is truly null, not the total metric count in a mixed batch.
What a proper fix would take: Simulating metrics with a realistic correlation structure instead of independent ones, and implementing the Benjamini-Yekutieli variant built for that case, would be the natural next step, along with letting the fraction of genuinely-lifted metrics vary within a single dashboard rather than being fixed for a whole run.