Superforecasting: The Art and Science of Prediction

One of the problems with forecasts is the ambiguity of their verbiage. Studies have shown that people differ in their interpretations of words like “significant” and “likely,” and often, those making the predictions do not delineate their definition. Many forecasts also lack concrete timescales. While forecasters might assume that a timescale is implied, failing to directly specify a timescale precludes measurement of the forecast’s accuracy.

After the disastrous 1961 Bay of Pigs invasion of Cuba by the Kennedy administration, history professor Sherman Kent devised a scale to address such discrepancies. Highly regarded in the intelligence community, Kent left Yale in 1941 to join the newly created Coordinator of Information agency (a precursor of the Central Intelligence Agency, or CIA). Kent argued that people needed to quantitatively define such terms such as “serious possibility,” and in 1964, he published a scale where 100% was certainty and 0% was impossibility (55). Other values, such as 75% (give or take about 12%), meant probable. While Kent’s scale aimed to reduce ambiguity, people were reluctant to adopt it, preferring their familiar linguistic methods to the numerical ones that made them feel bookish. Tetlock and Gardner argue that such numerical grading is useful; when forecasters are forced to break down their thinking processes, they become better at identifying more minute degrees of uncertainty, “just as artists get better at distinguishing subtler shades of gray” (57).

Still, the authors acknowledge that one downside to using numbers is what they call the “wrong-side-of-maybe fallacy,” whereby if a meteorologist predicts a 70% chance of rain, they also predict a 30% chance of no rain. This means that if it does not rain, their forecast would not necessarily be inaccurate. However, one way to predict with greater accuracy is to accrue several forecasts over time and then tabulate the results. For example, the accuracy of a particular meteorologist’s predictions can be calibrated within a specified period by using a graph with the x-axis showing the forecast and the y-axis showing the percentage correct. A perfect diagonal would indicate complete accuracy, while being above the diagonal line would indicate underconfidence and being below would indicate overconfidence.

Using a system developed by American statistician and meteorologist Glenn W. Brier in 1950, forecasters can be given a score that measures the distance between what they forecast and what actually happened. Zero is the perfect score, while 0.5 is the 50/50 guesswork of the dart-throwing chimpanzee from Chapter 1. However, it is important to consider Brier scores in context of the difficulty of the predictions made. Thus, a meteorologist scoring 0.2 in a changeable climate is more accurate than someone scoring the same in a predictable climate. The changeable-climate meteorologist is considered more accurate because they embody resolution as well as calibration, meaning that they do not merely play it safe in their predictions but instead use their data to make bold, supportable forecasts.

For Tetlock’s mid-1980s research program that he called Expert Political Judgment, he recruited 284 anonymous experts drawn from areas like academia and think tanks to make political predictions for events falling within specific time frames. Tetlock divided the experts into two groups, using Greek warrior-poet Archilochus’s distinction between people who are like hedgehogs and those who are like foxes. Archilochus judged that while hedgehogs know one big thing, foxes know lots of little things. The hedgehogs in Tetlock’s sample were big-idea people with politically diverse ideological bents. The hedgehogs tended to see all evidence through a lens that supported their core belief, even in the face of contradictory evidence. Hedgehogs’ singular perspective is a hindrance to forecasting because they are inflexible and prone to seeing evidence through their biases. Still, hedgehogs are often sought-after media presences because they can tell simple, provocative stories that captivate audiences.

In contrast, foxes, who know lots of little things and can see the uncertainty in situations, take their time to peruse contradictory data before reaching conclusions. This tendency can make foxes less charismatic media presences. Nevertheless, their deliberation makes foxes better forecasters. One key tool of foxes, both on an individual and a collective level, is aggregation, whereby they pool several judgments together before taking an average. Aggregation is particularly valuable when considering perspectives from many people who know different things that are relevant to the situation at hand. In the political sphere, aggregating data from different types of surveys, such as in the 2012 “poll of polls,” often leads to better electoral predictions.

The authors assert that aggregation, unlike “tip-of-your-nose thinking,” is counterintuitive and must be taught. While stepping outside of oneself and consciously seeking different perspectives can be difficult, foxes are more likely to do so than are hedgehogs. Still, Tetlock and Gardner acknowledge that people are not always consistent when it comes to predictions and decision-making and that people’s approaches can be more foxlike or more hedgehoglike depending on the context. However, the authors assert that people can become conscious of their tendencies and learn to switch gears between different thinking styles.