Some people seem to be better at forecasting future events than others.

One technique used by the superior forecasters is called “reference class forecasting.” Daniel Kahneman tells Joseph Walker the origin story of the concept in an interview on the Jolly Swagman podcast (“#143: Dyads, And Other Mysteries — Daniel Kahneman,” April 14, 2023, audio and transcript available). The hour-plus interview is quite interesting throughout: Kahneman must be one of the most interesting social scientists alive, and Walker does a nice job of drawing him out. On the topic of reference class forecasting, Kahneman says:

Well, first let’s define our terms, what the reference class is. I don’t know a better way of doing this than telling the origin story of that idea in my experience, which is that, approximately 50 years ago approximately, I was engaged in writing a textbook with a bunch of people at Hebrew University, a textbook for high school teaching of judgement and decision making. We were doing quite well, we thought we were making good progress. It occurred to me one day to ask the group how long it would take us to finish our job. There’s a correct way of asking those questions. You have to be very specific and define exactly what you mean. In this case I said, “Hand in a completed textbook to the Ministry of Education — when will that happen?” And we all did this. Another thing I did correctly, I asked everybody to do that independently, write their answer on a slip of paper, and we all did. And we were all between a year and a half and two and a half years.

But one of us was an expert on curriculum. And I asked him, “You know about other groups that are doing what we are doing. How did they fare? Can you imagine them in the state that we are at? How long did it take them to submit their book?” And he thought for a while, and in my story he blushed, but he stammered and he said, “You know, in the first place they didn’t all have a book at the end. About 40%, I would say, never finished. And those that finished…” He said, “I can’t think of any that finished in less than eight years — seven, eight years. Not many persisted more than ten.”

Now, it’s very clear when you have that story, that you have the same individual with two completely different views of the problem. And one is thinking about the problem as you normally do — thinking only of your problem. And the other is thinking of the problem as an instance of a class of similar problems.

In the context of planning, this is called reference class planning. That is, you find projects that are similar and you do the statistics of those projects, and it’s absolutely clear. It was evident to us at the time, but idiotically, I didn’t act on it. That was the correct answer, that we were 40% likely not to succeed. Because I also asked a friend, the curriculum expert, I asked “When you compare us to the others, how do we compare?” He said, “We are slightly below average.” So the chances of success were clearly very limited.

So that’s reference class forecasting. Now, how do you pick a reference class? In this case it was pretty obvious. I mean, we were engaged in creating a new curriculum. In other cases, when you are predicting the sales of the book or the success of the film, what is the reference class?

So if it’s a director and he’s had several films, is the reference class his films or similar films, same genre or whatever? And there isn’t a single answer, the answer is actually… You were asking how do you choose a reference class, my advice would be… And today I’m not an expert on that. The expert on that is Bent Flyvbjerg at Oxford. And I think what he probably would tell you is, “Pick more than one reference class to which this problem belongs.” Look at the statistics of all of them and if they are discrepant, you need to do some more thinking. If they all tend to agree, then you probably have got it more or less right.

The idea of reference class forecasting is applicable in many contexts: business plans, public policy implementation, outcomes of personal decisions, and others. It’s a way of getting outside your own head–outside the plans you can write down on a calendar or a spreadsheet about what you hope will happen. In thinking about the appropriate reference class and the experience of others with similar situations, you may also find that there are ways to improve your chances of success.