BIN There, Done That

¹ “Want Better Forecasting Skills? Silence the Noise,” Knowledge@Wharton, November 26, 2019.

² Philip E. Tetlock and Dan Gardner, Superforecasting: The Art and Science of Prediction (New York: Crown Publishers, 2015)

Anticipating revisions in market expectations is the key to generating long-term excess returns as an investor. Successful long-term investors see where expectations are headed based on their forecasts, but research shows that most experts do not make great forecasts.

Barbara Mellers, a professor of psychology at the University of Pennsylvania, says, “We find prediction really hard, but we find explanation fairly easy.”¹ We tell stories to ourselves and others to paper over our poor predictions. But the news is not all gloomy. The U.S. intelligence community sponsored a forecasting tournament, which revealed that one-in-fifty “superforecasters” consistently made better predictions than did the other participants.

Part of the success of the superforecasters reflected their qualities, including numeracy, intellectual curiosity, probabilistic thinking, and open-mindedness. But part of their success relied on habits that can be “learned and cultivated by any intelligent, thoughtful, determined person.”² 

Statisticians and psychologists examined the gap between “superforecasters” and regular forecasters using the “BIN Model,” where “B” refers to bias, “I” to information and “N” to noise.

When the researchers tallied why the superforecasters were so much better than the regular forecasters, they found the difference was roughly 50 percent from reducing noise, 25 percent from reducing bias and 25 percent from increasing information. It’s important to recognize that because noise and bias are independent sources of error, reducing either one of them improves forecasts.

When the researchers examined the impact of training on overall accuracy, they also found that the training improved accuracy more by reducing noise than by tamping down bias.

Noise. The three primary ways to reduce noise are combining judgments, using algorithms, and adopting the “Mediating Assessments Protocol” (MAP).

Combining forecasts is the core idea behind the wisdom of crowds, with the market a good example of a mechanism that aggregates views. Forecasting accuracy is compromised when there is no effective way to aggregate the information, or if the errors become correlated and noise transitions to collective bias.

Algorithms are simply sets of rules or procedures that allow you to achieve a goal. For example, a cake recipe is an algorithm. Research has found that one way to increase the acceptance of algorithms is to allow humans to adjust the final answer just a bit, what Daniel Kahneman calls “disciplined intuition.”

Finally, the Mediating Assessments Protocol (MAP) is a way to create intermediate ratings based on what’s critical in order to come to a final decision that is more informed than relying solely on intuition.

Bias. Debiasing methods introduce base rates to ground forecasts, premortems and red teams to open minds to alternative outcomes, and signposts to create a way to learn from mistakes. The methods produce spillover effects that help manage noise as well.

Information. Using information effectively means getting an informational advantage when you can, updating your view accurately and promptly to reflect new information, and placing the proper weight on the information at your disposal. Analysis of the forecasters in the tournament revealed that the frequency of belief updating was “the strongest single behavioral predictor of accuracy.”

Investors can use the BIN model to assess their decision-making process, with a heightened awareness of the large and potentially overlooked role of noise. Because noise tends to be overlooked or ignored, improving consistency can add a great deal of value, and identifying companies that adopt practices to reduce noise may be the source of excess returns.