“Where’s this special sauce?” CIA operative Bernadette in The Report asks psychologists James Mitchell and Bruce Jessen, after 60 days of waterboarding Muhammad Rahim yielded no results. “You have to make this work. It’s only legal if it works.”
Oracles, augurs, prophets, and pundits studied entrails, birds, and dreams looking for the special sauce, but the science of forecasting is young. A new article in the International Journal of Forecasting, which I wrote with colleagues Deniz Marti and Thomas Mazzuchi, claims to have found the special sauce—or at least some key ingredients—in the classical model for structured expert judgment. Torture isn’t among them.
Good expert forecasting does not correlate with citations, status, or blue ribbons. If you want to pick a good prophet, you look at her record. But her record of what? Tallying previous forecast errors isn’t as simple as it sounds. An error of three years in guessing the age of a Boeing-737 is not the same as an error of three years in guessing the age of your youngest daughter. Selecting forecasters based on previous errors requires converting errors to a common scale that accounts for the difference between aircraft years and daughter years.
Here’s the first key ingredient of the special sauce: the common scale is probability. How likely is an error of three aircraft years or three daughter years? That would be easy if we had a universal likelihood scale for all responses, but alas, we don’t. The next-best thing is the subjective probability of the expert forecaster herself. Suppose that for every forecast of a quasi-continuous unknown quantity, we tally how often the true value, revealed after the fact, falls within the expert’s 90 percent confidence bands.
Why would this help? Aren’t all experts pretty much equally able to assess how much they know? As it turns out, no. For some experts, saying they’re 90 percent certain that the true value falls in some interval confers a less than 10 percent chance of that predicted result actually happening. For other experts, you can take that 90 percent to the bank. Understanding an expert’s ability to gauge the likelihood of her own predictions might be good to know before we bet the farm on her advice.
Equipped with a common scale, we can reveal the second key ingredient of the special sauce: experts are not equal in their ability to quantify uncertainty. But how do we measure “the ability to quantify uncertainty,” and how we can use that information?
The third and perhaps most surprising sauce ingredient can help us: it’s better to combine expert uncertainties than to combine their point forecasts, and it’s better still to combine expert uncertainties based on their past performance.
"Sauce" - Google News
July 27, 2020 at 09:51PM
https://ift.tt/32XcEbf
The Science of Forecasting: The Special Sauce - Resources Magazine
"Sauce" - Google News
https://ift.tt/35DSBgW
Shoes Man Tutorial
Pos News Update
Meme Update
Korean Entertainment News
Japan News Update
Bagikan Berita Ini
0 Response to "The Science of Forecasting: The Special Sauce - Resources Magazine"
Post a Comment