How Wald Inferred Vital Areas from Returning Aircraft

According to the Wikimedia Commons description, this is a B-17 of the U.S. 398th Bomb Group whose nose was destroyed by flak during a bombing raid on Cologne, Germany, on October 15, 1944. The plane still made it back to base. Image: U.S. Air Force (National Museum of the U.S. Air Force), via Wikimedia Commons; license: public domain.

In 1943, Abraham Wald of Columbia University's Statistical Research Group (SRG) wrote a set of memoranda on how to estimate, using only the bullet holes on returning aircraft, the probability that a plane would be shot down after being hit in each of its parts. The popular story reduces this to a one-line quip: “reinforce wherever there are no holes.” Checked against the original memoranda and Mangel and Samaniego's 1984 interpretation, the actual reasoning was more subtle, and many of the details in circulation cannot be verified.

Columbia University's Statistical Research Group

The Statistical Research Group (SRG) was established on July 1, 1942, at Columbia University in New York. It was funded by the Applied Mathematics Panel (AMP) of the National Defense Research Committee (NDRC), which sat under the U.S. Office of Scientific Research and Development (OSRD). W. Allen Wallis, the group's director of research, recorded several figures in a 1980 retrospective article. Over its lifetime the group had 18 principal members and a total of 78 staff. At most about 50 people were on duty at once, including about 30 young women who worked only on computation. During the war the group produced 572 reports, memoranda and letters. Its members included Friedman, Savage, Mosteller, Wolfowitz and others who later became leading figures.

Wald was born in 1902 in Cluj, in present-day Romania. He worked on pure mathematics in Vienna in his early years and turned to mathematical statistics in the mid-thirties. Later, after losing his position in Austria because he was Jewish, he made his way to the United States. Wallis recalled that in the fall of 1939, when Wald had just begun teaching at Columbia, he and Wolfowitz often attended Wald's evening lectures. Wald was a part-time member of SRG and took part in its work for about 30 months in total. His best-known achievement there was sequential analysis, which originated at SRG in 1943. According to Wallis, the classification of sequential analysis as secret had a side effect: Wald's naturalization application was approved quickly, making him one of the few “enemy aliens” to obtain U.S. citizenship during the war.

Aircraft vulnerability was one of the topics that grew out of SRG's air-warfare research. In SRG's final report, written in 1945, Wallis wrote that this problem led the group to devise a method of “judging vulnerability from the damage sustained by our own planes.” Mangel and Samaniego summarize the operational problem of the time this way: commanders had to decide which tactics to use and which parts of the aircraft to give extra protection, but added armor increases weight and reduces a plane's ability to carry out its mission. The real difficulty was that no one could see how the bullet holes were distributed on the planes that failed to return.

The Reasoning and Numerical Example in the Memoranda

Wald's work took the form of a set of technical memoranda under the overall title “A Method of Estimating Plane Vulnerability Based on Damage of Survivors.” According to the Defense Technical Information Center's entry, these are eight memoranda that SRG wrote in 1943 for the National Defense Research Committee. They were not published during the war and were reprinted only in July 1980, by the Center for Naval Analyses (CNA), as CRC 432. The first few parts do not distinguish among parts of the aircraft. Using only the number of sorties, the number of planes that returned, and the number of hits on each returning plane, they derive the probability that a plane is downed by the i-th hit, along with upper and lower bounds on that probability. Part Five divides the aircraft into several “equal-vulnerability regions” and estimates the vulnerability of each part. Part Six deals with sampling error and gives 95% and 99% confidence limits.

The numerical example in Part Five shows the line of reasoning most clearly. Wald supposes that 400 planes fly a bombing mission and 359 return. Of the returning planes, 240 were not hit, while 68, 29, 12 and 10 planes took 1 to 4 hits respectively, for a total of 202 hits. He adopts simplifying assumptions, such as “every hit has the same probability of downing the plane, regardless of how many hits came before.” Under these assumptions he first solves for the probability that a plane survives a single hit, which comes to about 0.85. He then divides the aircraft into the engine, the fuselage, the fuel system and other parts. He assumes that the probability of a single hit landing on a given part equals that part's share of the projected area facing the direction of enemy attack: engine 26.9%, fuselage 34.6%, fuel system 15.4%, other parts 23.1%.

The hits on the returning planes, however, did not follow these proportions. The engine accounted for only 19.3% of them (39 hits), the fuselage 38.6% (78), the fuel system 15.4% (31), and other parts 26.7% (54). If a part's share of hits on surviving planes is lower than the share it “should” receive based on its area, then more of the planes hit there failed to return. By Wald's formula, the probability of surviving a single hit on a given part equals 0.85 multiplied by “actual share ÷ expected share.” That gives about 0.61 for the engine, about 0.95 for the fuselage, 0.85 for the fuel system, and about 0.98 for other parts. On this basis the memorandum states that, for the data of this hypothetical example, the engine area is the most vulnerable and the fuselage is relatively less vulnerable.

The hypothetical example in Part Five of Wald's memoranda. The engine's share of hits on returning planes is lower than the share its area would predict, and its estimated single-hit kill probability is the highest. Data from Wald's (1943) memoranda, CNA 1980 reprint; chart drawn with code for this article.

What Survivorship Bias Means and What It Assumes

This is what later came to be called survivorship bias: whether an object makes it into the sample depends on the very outcome we want to study. The returning planes are not a random sample of all the planes that flew. They are only the ones that “could still fly home after being hit.” If you count holes on them, the places with the most holes are often the places that can best withstand fire. In a short 1980 piece responding to readers' comments, Wallis put the point plainly. The military was inclined to add protection to the parts of returning planes that had been hit most often. Wald, however, had good reason to assume that hits in combat were spread roughly evenly over the whole aircraft. Hits on vital parts would therefore be underrepresented on the returning planes, because planes hit in those parts were less likely to come back and supply data.

Still, “reinforce where there are no holes” is only shorthand, and it holds only under one condition: we must first know how hits would have been distributed if no planes had been shot down. In Part Five, Wald specifically points out that the damage on returning planes alone cannot be used to estimate both the hit distribution and the vulnerability of each part. He gives an example. Suppose all the holes on returning planes fall in Region One. That could mean Region Two simply cannot be hit, or it could mean that any plane hit in Region Two goes down. The data cannot tell the two explanations apart. He therefore required the hit distribution to be known in advance. It could be inferred from the dispersion of enemy fire, or measured in firing tests with “dummy bullets.” In the numerical example, the proportions of projected area serve in its place.

In other words, Wald was not comparing “holes versus no holes.” He was comparing “how many fewer holes than expected.” In his example the engine still had 39 holes, just fewer than its area would predict. If a spot has few holes only because it is small and rarely hit, that is no reason to conclude that hits there are more deadly. In their 1984 article, Mangel and Samaniego re-examined these memoranda using modern statistical theory and discussed them within the framework of missing data and approximation methods. They concluded that Wald's treatment of these problems “was definitive.”

Exaggerations and Unsupported Details in Popular Versions

The popular version goes roughly like this. Looking at the pattern of holes on returning bombers, the military was about to reinforce the heavily hit fuselage. Wald pointed out on the spot that the sparsely hit engines were what needed reinforcing, and his advice was adopted. In 2016, the mathematician Bill Casselman checked this story point by point in his column for the American Mathematical Society. When it is compared with the original sources, its claims fall into three categories.

The documented parts: Wald did write a set of memoranda at SRG on inferring vulnerability from damage to surviving aircraft. Wallis's 1980 rejoinder did say that the military was inclined to reinforce the parts of returning planes that had been hit most often, and it stated that these methods were later used by the Navy and the Air Force. CNA's note to the reprint and Mangel and Samaniego's article both mention that the method was still used to analyze data from the Korean and Vietnam Wars.

The exaggerated parts: the memoranda are rigorous and highly technical mathematical derivations, and Casselman notes that they make no recommendation about what the military should do. The claim that “the engine is the most vulnerable” comes from the hypothetical data in Part Five. The text says explicitly that this conclusion applies to “this hypothetical example” and is not a finding measured in any actual campaign. Telling the story as a flash of insight also leaves out the assumptions Wald made to solve the identification problem, as well as the separate treatment he gave to sampling error.

The unsupported parts: no original record currently supports the conversations and meeting scenes between Wald and military officers, or the so-called “Wald's hand-drawn bullet-hole diagram” that circulates online. Casselman originally found that the only reliable material on this work was two brief passages in Wallis's retrospective article that do not mention Wald by name, plus the memoranda themselves. He later added Wallis's rejoinder in an addendum. The red-dotted airplane diagrams commonly seen online were all drawn by others later, and some even show a C-47 transport, a plane that rarely saw combat.

Chance also played a part in the memoranda becoming public nearly forty years late. Casselman quotes Phil DePoy of the Center for Naval Analyses on how this happened. When Wallis was moving house, he came across a box of loose papers left over from when SRG shut down in 1946 and handed it to DePoy. DePoy kept only one packet from it, Wald's material, edited it lightly, and published it under Wald's name in July 1980. Wald himself had died in 1950 in a plane crash in the mountains of southern India, and he left no recollections of his own about this work.

References

Wald, A. (1980). A Reprint of "A Method of Estimating Plane Vulnerability Based on Damage of Survivors" by Abraham Wald (CRC 432). Center for Naval Analyses. Original memoranda written in 1943. https://www.cna.org/reports/1980/0204320000.pdf
Defense Technical Information Center. Entry ADA091073. https://apps.dtic.mil/sti/citations/ADA091073
Mangel, M., & Samaniego, F. J. (1984). Abraham Wald's Work on Aircraft Survivability. Journal of the American Statistical Association, 79(386), 259–267. https://doi.org/10.1080/01621459.1984.10478038
Wallis, W. A. (1980). The Statistical Research Group, 1942–1945. Journal of the American Statistical Association, 75(370), 320–330. https://doi.org/10.1080/01621459.1980.10477469
Wallis, W. A. (1980). The Statistical Research Group, 1942–1945: Rejoinder. Journal of the American Statistical Association, 75(370), 334–335. https://doi.org/10.1080/01621459.1980.10477472
Casselman, B. (2016, June). The Legend of Abraham Wald. AMS Feature Column. https://www.ams.org/publicoutreach/feature-column/fc-2016-06
Research Methods · Rituals can be imitated; understanding cannot be faked
Compiled and published by Scholay, a one-stop academic platform. If this infringes your rights, please let us know and it will be taken down.
Also published on EnterScholar, a community for scholars in the age of intelligence