Where it sits
The dangerous part is that the missing data leaves no trace of being missing.
Tell it apart
| Aspect | Survivorship bias | Selection bias |
|---|---|---|
| Filter | Survival | Any |
| Scope | Narrow | Broad |
| Classic case | Wald's aircraft | Any unrepresentative sample |
| Relationship | A subtype | The category |
Survivorship bias is a subtype of selection bias — the specific case where the filter is survival or success. All survivorship bias is selection bias; not all selection bias involves anything surviving.
| Aspect | Survivorship bias | Confirmation bias |
|---|---|---|
| Where the error sits | In the data you can see | In the person weighing it |
| Classic case | Wald's returning aircraft | Reading only what agrees with you |
| Kind of bias | A statistical selection error | A habit of reasoning |
| First question to ask | What didn't make it into the sample? | What would change my mind? |
Survivorship bias is a fault in the evidence that reaches you: you only see the cases that passed some filter, the start-ups that lived or the planes that came home, and the failures are missing. Confirmation bias is a fault in how you handle evidence: you search for, notice and remember what fits what you already believe. One is a gap in the sample; the other is a slant in the reader. They stack easily, because survivors' stories are exactly what a believer goes looking for.
What it doesn't mean
It happened. Wald's eight memoranda were produced in 1943 for the National Defense Research Committee, and the full set, never published externally at the time, was reprinted by the Center for Naval Analyses in 1980. You can read it.
That's the punchline, not the work. Wald derived a mathematical method for estimating, part by part, the probability that a plane hit there would be lost — reasoning backwards from the damage on survivors to the damage on everything that flew. The famous red-dotted bomber diagram is a modern illustration, not his figure.
What it really means
The military brought Abraham Wald a genuinely reasonable question in 1943: given a map of where returning aircraft had been hit, where should the extra armour go? The bullet holes clustered on the wings and fuselage. The obvious answer was to armour the wings and fuselage.
Wald's answer was to armour the places with no holes. The data covered planes that came back. A plane hit in the engine did not come back, and so contributed no dot to the map. The blank regions weren't areas that never got shot — they were areas where getting shot was fatal, and the silence was the finding.
Worth correcting the version that circulates online: Wald did not glance at a diagram and deliver a quip. Across eight memoranda he built a mathematical method for estimating, component by component, the probability that a hit there would down the aircraft — reconstructing the full population from the survivors. The famous red-dotted bomber picture is a modern illustration, not his.
What makes survivorship bias so durable is that the missing data leaves no trace of being missing. Nothing on the map said engines untested. The sample looked complete, because absent evidence never announces itself — which is the same reason a shelf of founder memoirs looks like a study of what works.
Where it comes from
borrowed from actuarial and finance usage
In finance, a 'survivorship-biased' index is one that quietly drops the funds that collapsed.
- 1943
Abraham Wald, working at Columbia's Statistical Research Group, writes eight memoranda on aircraft vulnerability. The available data recorded where returning aircraft had been hit — a sample that excluded every plane shot down.
- 1980
Wald's wartime memoranda, never published externally during the war, are reprinted by the Center for Naval Analyses. The method had by then been used on damage data from Korea and Vietnam.
In real life
- They don't build them like they used to, says someone admiring a 200-year-old cottage. The badly built cottages of the same era fell down long ago and aren't there to compare.
- Investment funds that do badly are often closed or merged into other funds, so a list of today's funds is a list of the ones that survived, and their average record looks better than the real one.
- A famous founder who dropped out of university becomes the proof that dropping out works. The many dropouts whose start-ups failed never give a talk.
- A 1987 study found that cats falling from higher than six storeys were injured less than cats falling from lower floors. One suggested explanation is survivorship: cats killed outright are less likely to be taken to a vet, so they never reach the data.
- Every bestselling author interviewed says they never gave up. Nobody interviews the equally stubborn writers who never got published.
Questions
Q01What is survivorship bias?
Studying only the winners. Five biographies of college dropouts who founded billion-dollar companies tell you nothing useful about dropping out, because the thousands who dropped out and failed never got a book deal.
Q02What did Abraham Wald actually do?
In 1943 he worked from data on where returning aircraft had been hit, and produced a method for estimating the vulnerability of each part of a plane from that survivor-only data — the areas showing least damage among survivors being the ones most likely to be fatal. The familiar version, in which he personally stops engineers from armouring the bullet holes and names the engines, is a later embellishment.
Q03Where does it show up in everyday life?
Reviews, mostly. A restaurant's page shows the customers who came back to post; the ones who had a bad night simply never returned and never wrote anything. A 4.6 average can be an average of people already inclined to like it.
Sources
- A Method of Estimating Plane Vulnerability Based on Damage of Survivors — Center for Naval Analyses.
- DTIC ADA091073 — full document record, Internet Archive.
- survivorship (etymology) — Wiktionary.
- Survivorship bias — Wikipedia.
- Confirmation bias — Wikipedia.