Berkson's Paradox

The modern name for Berkson's paradox is collider bias, and the name comes from the picture. Draw the causal graph:

$$A \longrightarrow H \longleftarrow B$$

Two arrows collide at $H$ — hospitalisation, or being hired, or being in your dating pool. $H$ is a common effect of $A$ and $B$.

This is the third of the three elementary structures a variable can sit in, and it behaves in the opposite way from the other two.

• Chain $A \to M \to B$: $A$ and $B$ are associated; conditioning on $M$ breaks the association.
• Fork $A \leftarrow C \to B$: $A$ and $B$ are associated; conditioning on $C$ breaks the association.
• Collider $A \to H \leftarrow B$: $A$ and $B$ are not associated; conditioning on $H$ creates an association.

The first two match intuition — conditioning cleans things up. The collider does not, and this is the single most common source of error in observational research. Judea Pearl's contribution was to make the rule mechanical: given a graph, an algorithm decides which variables you must adjust for and which you must not touch. The reflex "control for everything you measured" is wrong, and this is why. Adjusting for a collider does not add noise or waste power; it introduces bias where there was none.

A subtlety worth stating: conditioning on a descendant of a collider produces the same effect, in weaker form. You do not need to select on hospitalisation itself. Selecting on anything downstream of it — having a hospital record, being reachable through a clinic mailing list, having answered a survey sent to patients — leaks the same bias in.

Is this really the same thing as Simpson's paradox?

They are close cousins and are frequently conflated, but they are not the same, and the difference is exactly the one that matters for what you should do.

Simpson's paradox is usually generated by a confounder — a fork — and the fix is to condition on it. Berkson's paradox is generated by a collider, and conditioning on it is the disease rather than the cure.

The two look alike because both are cases where a table changes its story when you slice it. They come apart because in one case slicing is the right move and in the other it is precisely the wrong one — and the data cannot tell you which case you are in. That is the same lesson from both directions: conditioning is a causal act, and it needs a causal justification.

How big can it get?

Berkson's own view was that the effect was large enough to explain published findings, not merely to add noise. Sackett's 1979 catalogue of biases went further and gave empirical support: he showed hospitalisation rates that differ enough between conditions for the artefact to bite in practice, and named it "admission rate bias" in his list of thirty-five ways clinical research goes wrong.

There is also a strand of criticism, which deserves its due, arguing that Berkson bias is often invoked more readily than the numbers warrant — that in many real settings the differential admission rates required to produce a large artefact are implausibly extreme. The honest position is that the artefact's size depends on how strongly each factor drives selection, that this is measurable in principle, and that it is usually not measured. Quantitative bias analysis exists to fill that gap, and remains under-used.

1946. Joseph Berkson, head of biometry at the Mayo Clinic, published a short note in Biometrics Bulletin under a deliberately dry title: "Limitations of the Application of Fourfold Table Analysis to Hospital Data". His concern was practical and immediate. Hospital records were the cheapest source of data on disease association, and researchers were building two-by-two tables from them as though the patients were a sample of the public. Berkson worked through the arithmetic for cholecystitis and diabetes and showed that a spurious association — of either sign, and of substantial size — could be generated by admission rates alone.

Berkson is a complicated figure. He was a brilliant statistician, coined the term "logit", and was also, for the rest of his career, one of the most persistent opponents of the conclusion that smoking causes lung cancer. Some of his objections in that debate were versions of this very argument: hospital-based case-control studies, he maintained, were exactly the kind of design his 1946 paper had warned about. He was wrong about smoking, and the epidemiologists who answered him — Cornfield above all — developed sensitivity analyses that showed no plausible selection effect could account for the observed risk ratios. The paradox that bears his name is sound; his most famous application of it was not.

1955–1980. The result was rediscovered and renamed repeatedly across fields — "Berkson's fallacy", "admission rate bias", "selection bias", and in psychometrics "restriction of range", which is the same structure applied to continuous traits. David Sackett's 1979 catalogue fixed the epidemiological terminology.

1988 onwards. Judea Pearl's work on graphical models supplied the general account. Once causal structure is drawn as a graph, Berkson's paradox is not a special hazard of hospital data but one of exactly three ways that conditioning changes an association — the collider case. The d-separation criterion turns Berkson's careful hand-worked warning into a rule a computer can check.

2000s. The term "collider bias" entered general epidemiological use, largely through Miguel Hernán, Sander Greenland, Sonia Hernández-Díaz and colleagues, who showed that a striking number of published puzzles — including the "birthweight paradox" and several obesity-mortality reversals — had this shape.

Berkson's paradox is the reason that "we analysed a very large clinical database" is not, by itself, reassuring.

It reframed study design. Case-control studies drawing controls from hospital wards were standard practice; after 1946 they needed a defence, and the modern insistence that controls be drawn from the same source population as the cases is a direct consequence. The whole apparatus of thinking carefully about the sampling frame — rather than only the sample size — descends from this.

It broke the "adjust for everything" heuristic. For decades, adding covariates to a regression was treated as unambiguously conservative: it could cost you precision, never validity. The collider case shows that is false. A variable affected by both your exposure and your outcome will bias the estimate if you include it, and the bias can be larger than the effect you are hunting. Modern practice — draw the assumed graph, then let the graph choose the adjustment set — exists because of this.

It explains a durable class of false beliefs about the world. The perception that talent and social skill are in tension, that beauty and intelligence trade off, that successful companies must have had luck rather than merit, that high-scoring students are less creative: each of these is a plausible reading of a selected sample, and each survives because the counterexamples are systematically invisible. Not everything that looks like a trade-off is one. Some of them are just doors.

COVID-19 risk factors. Griffith and colleagues showed in 2020 that early studies drawing on volunteer cohorts and on tested-only samples were badly exposed to this bias, because both the risk factors and the outcome affected who got tested at all. In their analysis the direction of well-established associations could invert — smoking, for one, appearing protective in test-selected data. This is Berkson's 1946 warning arriving in the middle of a pandemic, at scale, with policy attached.

The obesity paradox. Several studies have reported that among patients who already have heart failure, diabetes or kidney disease, higher body mass index predicts better survival. One leading explanation is collider bias: you are conditioning on having the disease, and both obesity and other unmeasured causes drive people into that group, so the obese patients are systematically less burdened by the other causes. The debate is not fully settled, which is part of why it is a good illustration — the artefact is hard to rule out, and hard to rule in.

Recommender systems and feedback loops. A recommender only observes outcomes for items it chose to show. Whether a user would have liked an unrecommended item is unobservable, and the training data is selected on a variable downstream of both the item's quality and the user's taste. Off-policy evaluation, inverse propensity weighting and deliberate exploration are all machinery for undoing exactly this.

Machine learning on convenience data. Any model trained on records of people who reached a particular system — loan applicants who were approved, patients who were admitted, users who completed onboarding — learns relationships that hold in that slice. Deployed on the general population, the learned relationship may not just weaken; it may point the wrong way. "Distribution shift" covers many sins, and this is one of the sharpest.

Scientific literature itself. Journals select papers on some mix of novelty and rigour. Among published papers, the two will appear negatively correlated, which is a decent account of the widespread impression that the most exciting findings are the least solid. The impression is real. The trade-off is the editorial threshold.