Reference Class Forecasting: An Origin Story from Daniel Kahneman

Reference Class Forecasting: An Origin Story from Daniel Kahneman

Reference Class Forecasting: An Origin Story from Daniel Kahneman

Some people seem to be better at forecasting future events than others.

One technique used by the superior forecasters is called “reference class forecasting.” Daniel Kahneman tells Joseph Walker the origin story of the concept in an interview on the Jolly Swagman podcast (“#143: Dyads, And Other Mysteries — Daniel Kahneman,” April 14, 2023, audio and transcript available). The hour-plus interview is quite interesting throughout: Kahneman must be one of the most interesting social scientists alive, and Walker does a nice job of drawing him out. On the topic of reference class forecasting, Kahneman says:

Well, first let’s define our terms, what the reference class is. I don’t know a better way of doing this than telling the origin story of that idea in my experience, which is that, approximately 50 years ago approximately, I was engaged in writing a textbook with a bunch of people at Hebrew University, a textbook for high school teaching of judgement and decision making. We were doing quite well, we thought we were making good progress. It occurred to me one day to ask the group how long it would take us to finish our job. There’s a correct way of asking those questions. You have to be very specific and define exactly what you mean. In this case I said, “Hand in a completed textbook to the Ministry of Education — when will that happen?” And we all did this. Another thing I did correctly, I asked everybody to do that independently, write their answer on a slip of paper, and we all did. And we were all between a year and a half and two and a half years.

But one of us was an expert on curriculum. And I asked him, “You know about other groups that are doing what we are doing. How did they fare? Can you imagine them in the state that we are at? How long did it take them to submit their book?” And he thought for a while, and in my story he blushed, but he stammered and he said, “You know, in the first place they didn’t all have a book at the end. About 40%, I would say, never finished. And those that finished…” He said, “I can’t think of any that finished in less than eight years — seven, eight years. Not many persisted more than ten.”

Now, it’s very clear when you have that story, that you have the same individual with two completely different views of the problem. And one is thinking about the problem as you normally do — thinking only of your problem. And the other is thinking of the problem as an instance of a class of similar problems.

In the context of planning, this is called reference class planning. That is, you find projects that are similar and you do the statistics of those projects, and it’s absolutely clear. It was evident to us at the time, but idiotically, I didn’t act on it. That was the correct answer, that we were 40% likely not to succeed. Because I also asked a friend, the curriculum expert, I asked “When you compare us to the others, how do we compare?” He said, “We are slightly below average.” So the chances of success were clearly very limited.

So that’s reference class forecasting. Now, how do you pick a reference class? In this case it was pretty obvious. I mean, we were engaged in creating a new curriculum. In other cases, when you are predicting the sales of the book or the success of the film, what is the reference class?

So if it’s a director and he’s had several films, is the reference class his films or similar films, same genre or whatever? And there isn’t a single answer, the answer is actually… You were asking how do you choose a reference class, my advice would be… And today I’m not an expert on that. The expert on that is Bent Flyvbjerg at Oxford. And I think what he probably would tell you is, “Pick more than one reference class to which this problem belongs.” Look at the statistics of all of them and if they are discrepant, you need to do some more thinking. If they all tend to agree, then you probably have got it more or less right.

The idea of reference class forecasting is applicable in many contexts: business plans, public policy implementation, outcomes of personal decisions, and others. It’s a way of getting outside your own head–outside the plans you can write down on a calendar or a spreadsheet about what you hope will happen. In thinking about the appropriate reference class and the experience of others with similar situations, you may also find that there are ways to improve your chances of success.

Share this article

Leave your comments

Post comment as a guest

0
terms and condition.
  • No comments found

Share this article

Timothy Taylor

Global Economy Expert

Timothy Taylor is an American economist. He is managing editor of the Journal of Economic Perspectives, a quarterly academic journal produced at Macalester College and published by the American Economic Association. Taylor received his Bachelor of Arts degree from Haverford College and a master's degree in economics from Stanford University. At Stanford, he was winner of the award for excellent teaching in a large class (more than 30 students) given by the Associated Students of Stanford University. At Minnesota, he was named a Distinguished Lecturer by the Department of Economics and voted Teacher of the Year by the master's degree students at the Hubert H. Humphrey Institute of Public Affairs. Taylor has been a guest speaker for groups of teachers of high school economics, visiting diplomats from eastern Europe, talk-radio shows, and community groups. From 1989 to 1997, Professor Taylor wrote an economics opinion column for the San Jose Mercury-News. He has published multiple lectures on economics through The Teaching Company. With Rudolph Penner and Isabel Sawhill, he is co-author of Updating America's Social Contract (2000), whose first chapter provided an early radical centrist perspective, "An Agenda for the Radical Middle". Taylor is also the author of The Instant Economist: Everything You Need to Know About How the Economy Works, published by the Penguin Group in 2012. The fourth edition of Taylor's Principles of Economics textbook was published by Textbook Media in 2017.

   
Save
Cookies user prefences
We use cookies to ensure you to get the best experience on our website. If you decline the use of cookies, this website may not function as expected.
Accept all
Decline all
Read more
Analytics
Tools used to analyze the data to measure the effectiveness of a website and to understand how it works.
Google Analytics
Accept
Decline