I love it when someone makes lots of predictions and then we can see their track record after the fact. Here's a collection of links to track records for anyone/anything interesting that has them tabulated, AFAIK.

The basic idea

The basic idea is that someone can write down specific, public predictions about the world with probabilities attached, like "60% chance that X wins the 2020 election." If they make a lot of predictions, they can then come back and assess whether their overall body of work was "well calibrated," which means that things they thought would happen 60% of the time happened 60% of the time; things they thought would happen 80% of the time happened 80% of the time; etc.

This is done using a "calibration curve" (most of the links below explain how the curve works; this explanation is pretty compact).

People with good historical calibration then have evidence of their trustworthiness going forward.

More on this general idea in the book Superforecasting (and more briefly at this Open Philanthropy blog post).

Good track records

• FiveThirtyEight
• HyperMind, which contracts to create predictions.
• A few individuals:
  • Scott Alexander (this links to year-by-year track records, though I wish all the years were combined in one place).
  • @peterhurford Twitter predictions (and summary).
  • Gwern.

OK-to-pretty-good track records

• ElectionBettingOdds.com, which forecasts elections based on prediction markets (unfortunately not updated for 2020 elections yet). Good overall, though it looks like events they score as 20-30% likely are more likely to happen than predicted. (I wish they would combine the 20-30% predictions with the 70-80% predictions etc., since a 20-30% prediction that something will happen is just like a 70-80% prediction that it won't.)
• Metaculus, a community forecasting site - skip to the last chart, which is the same idea as the 2nd chart from ElectionBettingOdds, though harder to read. Seems to be biased in the opposite direction from ElectionBettingOdds.com, i.e., overrating the likelihood of pretty unlikely events.
• All users in aggregate for PredictionBook (a website that lets individuals track their own predictions) - well calibrated except at the very confident end. I'd guess this is a "wisdom of crowds" thing; it wouldn't make me trust a particular PredictionBook user very much.

Less good track records

Track record for Scott Adams, scored by someone other than Scott Adams (so not 100% reliable, though I checked a couple of them). It's really bad, probably easier to see from the "Predictions" sheet than from the chart. For example he tweeted "I put Trump’s odds of winning the election at 100% (unless something big changes) with a 50% chance of a successful coup" on 6/18/2020, and "If I had to bet today, Trump has the edge, 60-40" two days after the election. (He also gave 0% to Biden winning the nomination, earlier.) The reason I'm bothering with this (and probably the reason the maker of the spreadsheet did) is that Adams has gotten attention in some circles for his highly confident predictions of a Trump win in 2016, and I want to make the general point that a small # of impressive predictions can be very misleading, since the successful predictions tend to get more attention than the unsuccessful predictions.

2015 evidence that a famous ESPN sports analyst had actually been editing his prediction-like statements (mock draft rankings) to make them look better with the benefit of hindsight.

Tangentially related

Independent analysis of Ray Kurzweil's predictions about 2019, made in 1999. He didn't give probabilities, and got more than twice as many wrong as right, though I think it's fair of the author to end with "I strongly suspect that most people's 1999 predictions about 2019 would have been a lot worse."

Open Philanthropy's attempt to assess the general accuracy of very long-range forecasts mostly concluded that it's too hard to assess because of things like "People didn't make clear enough predictions or give probabilities."

Subscribe Feedback