Simpson's Paradox

The question the paradox forces is: which table should you act on?

There is a widespread but wrong answer, sometimes taught as a rule: always disaggregate, because more detail is more truth. It is wrong because you can always keep splitting. Split Berkeley by department, then by subfield, then by year, then by the applicant's undergraduate institution. Each split gives new numbers, and the direction can flip more than once on the way down. If finer is always better, you end up conditioning on so much that every cell holds one person and comparison becomes meaningless.

Judea Pearl's diagnosis, developed through the 1990s and set out in Causality, is that the question is not answerable from the data at all. Two datasets can be numerically identical, cell for cell, and require opposite decisions — because the variables mean different things. Simpson himself made exactly this point in 1951, using two examples with the same arithmetic and different correct readings. He is often remembered as the man who found the paradox; he is better remembered as the man who said the numbers do not decide.

What decides is the causal structure. Three cases:

• The splitting variable is a common cause of both the treatment and the outcome — a confounder. Stone size affects which operation the surgeon chooses and how likely the patient is to recover. Split the table. The split table is right.
• The splitting variable sits between the treatment and the outcome — a mediator. If a drug lowers heart-attack risk by lowering blood pressure, then splitting by blood pressure hides the drug's effect. Do not split. The pooled table is right.
• The splitting variable is a common effect of both — a collider. Splitting on it manufactures an association from nothing. This is Berkson's paradox, and here splitting is not merely unhelpful, it is actively misleading.

Berkeley is genuinely contested precisely because department is arguable. Is department a confounder — women happen to be interested in different fields, for reasons outside the university's control? Or is it a mediator — women were steered towards, or pushed out of, particular fields by the same system that then admitted them? On the first reading, adjusting for department is correct and the university is exonerated. On the second, adjusting for department conceals part of the effect you were trying to measure, and the aggregate figure is the honest one. Bickel and his co-authors saw this clearly and said so: the data settle the question of committee bias and leave the larger question open.

The sure-thing principle

Leonard Savage, in The Foundations of Statistics (1954), stated a principle that sounds beyond dispute:

If you would prefer act $A$ to act $B$ if you knew that $E$ obtained, and you would also prefer $A$ to $B$ if you knew that $E$ did not obtain, then you should prefer $A$ to $B$ without knowing anything about $E$.

Savage illustrated it with a businessman deciding whether to buy a property before an election. He buys if the Democrat wins; he buys if the Republican wins; therefore he should buy now, without waiting for the result.

Simpson's paradox looks like a direct counterexample, and Colin Blyth pressed exactly this point in the paper that gave the effect its name in 1972. Open surgery is better for small stones and better for large stones, yet worse overall. Is that not "better whichever way $E$ turns out, and worse regardless"?

No — and the reason is the heart of the matter. Savage's principle is about acts. It requires that choosing $A$ rather than $B$ does not change whether $E$ obtains. In the kidney-stone data, choosing the treatment is entangled with the stone size, not because treatment causes stone size but because stone size caused the treatment. The event you are conditioning on is not independent of the choice you are evaluating.

Pearl's resolution is that the sure-thing principle is valid — once it is stated causally, in terms of interventions rather than observed frequencies. If a surgeon decides to operate openly on every small stone and decides to operate openly on every large stone, open surgery does win overall. The reversal lives entirely in the gap between "patients who received this treatment" and "patients assigned this treatment". Randomised trials exist to close that gap, which is another way of saying that randomisation is a machine for making the pooled table trustworthy.

1899–1903. Karl Pearson, with Alice Lee and Leslie Bramley-Moore, noted a "spurious correlation" produced by mixing heterogeneous records. George Udny Yule set the effect out sharply in Biometrika in 1903, in his notes on the theory of association of attributes, and treated it as a fallacy of illegitimate pooling. Everything essential is already there.

1951. Edward H. Simpson, a British statistician who had worked at Bletchley Park, published four pages in the Journal of the Royal Statistical Society on the interpretation of interaction in contingency tables. His contribution was not the arithmetic, which he took as known, but the interpretive claim: he gave two examples with identical numbers — one about a card trick, one about a medical treatment — and argued that the "sensible" reading differs between them. The tables cannot tell you which reading is sensible. That is the modern point, made in 1951.

1972. Colin R. Blyth, writing in the Journal of the American Statistical Association, called the effect "Simpson's paradox" and presented it as an apparent violation of Savage's sure-thing principle. The name stuck, to Simpson's mild embarrassment; he had been explicit that he was not claiming priority. Some statisticians still write "the Yule–Simpson effect" for accuracy.

1975. Peter Bickel, Eugene Hammel and J. William O'Connell published the Berkeley admissions analysis in Science. The Graduate Division had asked for it because the aggregate figures looked like grounds for a lawsuit. The paper is a model of restraint: it shows the committee-level bias is small and, if anything, favours women, and it says plainly that this leaves untouched the question of why women applied where they did.

1986–1994. Charig and colleagues reported kidney-stone outcomes in the BMJ. Julious and Mullee returned to the figures in 1994 under the title "Confounding and Simpson's paradox", and the example entered the medical curriculum, where it has stayed.

1990s–2000s. Judea Pearl's work on graphical causal models supplied the resolution that had been missing since Yule: a formal criterion, the back-door criterion, that says when to adjust and when not to, and a proof that the criterion cannot be read off the data.

Simpson's paradox is the reason "we looked at the data" is not an argument.

It changed how comparisons are reported. Age-standardised mortality rates, risk-adjusted surgical outcomes, and school value-added scores all exist because raw rates are known to be uninterpretable when the underlying populations differ. A hospital that takes the sickest patients will look worse than a hospital that turns them away, on every unadjusted measure, forever. Publishing unadjusted league tables does not just mislead; it creates an incentive to refuse difficult cases.

It supplies the strongest practical argument for randomisation. The whole point of assigning treatment by coin flip is to force the allocation weights to be equal across arms, which is exactly the condition under which the reversal cannot occur. Before Simpson's paradox is made vivid, randomisation looks like statistical fussiness; afterwards it looks like the only defence available.

It set a limit on what data alone can do. Pearl's demonstration that identical datasets can require opposite decisions is a genuine impossibility result about inference: causal conclusions require causal premises. This is now the standard framing across epidemiology, econometrics and machine learning, and it is why serious empirical papers now draw a diagram of assumed mechanisms before they report a coefficient.

And it is a permanent caution about discrimination statistics. The Berkeley case is cited in both directions — as proof that apparent bias can evaporate, and as proof that evaporating it can hide the real mechanism. Both citations are fair. The honest conclusion is narrower and more useful than either: the admissions committees were not the problem, and something upstream was.

Algorithmic fairness. A model can be calibrated within every subgroup and still show large disparities in aggregate, or the reverse. Several of the best-known impossibility results in fairness — the demonstration that calibration, equal false-positive rates and equal false-negative rates cannot generally hold together when base rates differ — are Simpson-shaped: the subgroup and the aggregate answer different questions and cannot be reconciled by choosing a better threshold.

A/B testing. A test can win overall and lose on desktop and lose on mobile, if the traffic mix shifted mid-experiment. Sample-ratio mismatch alarms exist in experimentation platforms for precisely this reason: unequal weights across arms are the precondition for the reversal, so a platform that detects them is detecting the paradox before it happens.

COVID-19 case fatality rates. Early in the pandemic, several countries reported higher raw case fatality rates than others while having lower rates in every age band — because their populations, or their tested populations, were older. In 2020 the same reversal appeared inside single countries when vaccinated and unvaccinated death counts were compared without age stratification: because the vaccinated group skewed heavily elderly, unadjusted figures could make vaccination appear ineffective while every age band showed a large benefit.

Salary and pay-gap analysis. Adjusting a gender pay gap for job title, seniority and department shrinks it — but only if those variables are confounders. If people were sorted into titles and departments by the same process under examination, they are mediators, and adjusting for them subtracts the effect from itself. Which they are is not a statistical question, and the argument about pay gaps is very largely an argument about this.

Model evaluation in machine learning. Aggregate accuracy is a weighted average over a test set's composition. A model that beats a baseline on every identifiable slice can lose on the headline metric, and vice versa, whenever the slice proportions in the test set differ from deployment. This is why sliced evaluation is now standard practice, and why a single leaderboard number is treated with suspicion.

The arithmetic

The reversal is a failure of the mediant to respect the ordering of fractions. For positive denominators, the pooled rate of two groups is

$$\frac{a_1 + a_2}{n_1 + n_2}$$

which is not the average of $a_1/n_1$ and $a_2/n_2$ but a weighted average:

$$\frac{a_1 + a_2}{n_1 + n_2} ;=; w_1 \cdot \frac{a_1}{n_1} ;+; w_2 \cdot \frac{a_2}{n_2}, \qquad w_i = \frac{n_i}{n_1 + n_2}$$

Two groups, two treatments. Let $p_z$ be treatment $A$'s success rate in stratum $z$ and $q_z$ treatment $B$'s, with strata weights $u_z$ for $A$ and $v_z$ for $B$ (each summing to 1). The reversal is the simultaneous truth of

$$p_1 > q_1, \qquad p_2 > q_2, \qquad \sum_z u_z p_z < \sum_z v_z q_z$$

If $u_z = v_z$ for every $z$ — the same mix on both sides — no reversal is possible, since a weighted average with fixed weights is monotone in its arguments. The reversal is entirely a product of unequal weights. That is why randomisation kills it: randomisation forces $u = v$ in expectation.

How far can it go? The gap you can hide is bounded by the spread of the stratum rates. Writing $\bar p = \sum u_z p_z$, the difference between the pooled comparison and the honest one decomposes as

$$\underbrace{\sum_z u_z p_z - \sum_z v_z q_z}{\text{pooled difference}} = \underbrace{\sum_z u_z (p_z - q_z)}{\text{genuine effect}}

• \underbrace{\sum_z (u_z - v_z), q_z}_{\text{mix effect}}$$

The second term is a covariance between allocation and baseline quality. It has no upper bound in the direction that matters: with two strata you can make the genuine effect arbitrarily small and positive while the mix effect is close to $q_{\max} - q_{\min}$. Blyth built his 1972 example this way, driving the reversal almost to the limit — which is the point, because it shows the effect is not a small-sample artefact that more data will wash out. More data makes a Simpson reversal more precise, not less.

Check it against Berkeley. Men's weights across A–F are heavily loaded onto A and B, which admit at 62% and 63%: $u_A = 825/2691 = 0.307$, $u_B = 0.208$, so over half the male applications went to the two most generous departments. Women's weights are $v_A = 108/1835 = 0.059$ and $v_B = 0.014$ — under 8% between them — while $v_C + v_D + v_E + v_F = 0.927$ lands in departments admitting 24% to 35%.

Now hold the mix fixed. Apply the men's department-by-department admission rates to the women's distribution of applications:

$$\sum_z v_z, p_z^{\text{men}} = 0.0589(0.621) + 0.0136(0.630) + 0.3232(0.369)

• 0.2044(0.331) + 0.2142(0.277) + 0.1858(0.059) = 0.302$$

Men facing the departments women actually applied to would have been admitted at 30.2%. Women were admitted at 30.4%. A fourteen-point gap becomes a fifth of a point, pointing the other way. Nothing was removed from the data except the difference in where the applications went.

The causal resolution

Let $X$ be treatment, $Y$ outcome, $Z$ the splitting variable. The two tables correspond to two different quantities:

• Pooled: $P(y \mid x)$ — the outcome among those who had $x$.
• Split, then recombined: $\sum_z P(y \mid x, z), P(z)$ — the standardised, or adjusted, rate.

Neither is automatically the causal quantity you want, which Pearl writes with the do-operator:

$$P(y \mid do(x))$$

— the outcome distribution if $X$ were set to $x$ by intervention, leaving everything upstream of $X$ untouched. The two observational quantities each equal $P(y \mid do(x))$ under different conditions, and never both at once when a reversal is present.

The back-door criterion says exactly when the adjusted formula is the causal one. A set $Z$ is admissible for the effect of $X$ on $Y$ if

1. no node in $Z$ is a descendant of $X$, and
2. $Z$ blocks every path from $X$ to $Y$ that starts with an arrow into $X$.

When $Z$ satisfies it, the adjustment formula holds:

$$P(y \mid do(x)) ;=; \sum_z P(y \mid x, z), P(z)$$

and the split table, recombined with the population weights $P(z)$, is correct.

Run the three graphs.

Confounder. $Z \to X$, $Z \to Y$, $X \to Y$. Stone size determines the surgeon's choice and the chance of success. There is a back-door path $X \leftarrow Z \to Y$; conditioning on $Z$ blocks it; $Z$ is not a descendant of $X$. Both conditions hold, so adjust. Standardising the kidney-stone data to the overall size distribution (357 small, 343 large):

$$P(\text{success} \mid do(\text{open})) = 0.93 \cdot \tfrac{357}{700} + 0.73 \cdot \tfrac{343}{700} \approx 0.83$$

$$P(\text{success} \mid do(\text{PCNL})) = 0.87 \cdot \tfrac{357}{700} + 0.69 \cdot \tfrac{343}{700} \approx 0.78$$

Open surgery wins, by about five points, in the direction the stratified table already indicated. The unadjusted 78% versus 83% is a fact about surgeons' triage decisions, not about surgery.

Mediator. $X \to Z \to Y$. Here $Z$ is a descendant of $X$, so the first back-door condition fails outright and the adjustment formula does not give the causal effect. Conditioning on a mediator removes precisely the part of the effect you are trying to measure. The correct answer is the pooled table.

Collider. $X \to Z \leftarrow Y$. The path $X \to Z \leftarrow Y$ is blocked by default; conditioning on $Z$ unblocks it and creates dependence where none existed. Adjusting is worse than useless. See Berkson's paradox.

The conclusion is the one that makes statisticians uncomfortable and is nevertheless right: the data are consistent with every one of these graphs. $P(x, y, z)$ is the same in all three. The choice of table is underdetermined by any amount of observation, and is fixed only by assumptions about the mechanism — assumptions that must be argued for, not estimated.

Formally, Pearl's sure-thing principle recovers Savage: if $P(y \mid do(x), z) > P(y \mid do(x'), z)$ for every $z$, and $Z$ is not affected by $X$, then $P(y \mid do(x)) > P(y \mid do(x'))$. There is no interventional Simpson's paradox. The reversal is a purely observational phenomenon, and it should be read as a signature that something is standing between you and the causal question.