Bias Has Been Overestimated at the Expense of Noise

Bias Has Been Overestimated at the Expense of Noise

Timothy Taylor 11/04/2019 5
Daniel Kahneman (Nobel 2002) is of course known for his extensive work on behavioral biases and how they affect economic decisions. He's now working on a new book, together with Olivier Sibony and Cass Sunstein, in which he focuses instead on the concept of "noise".
Here's a précis of Kahneman's current thinking on this and other topics, drawn from an interview with Tyler Cowen (both video and a transcript are available at "Daniel Kahneman on Cutting Through the Noise," December 19, 2018).

KAHNEMAN: First of all, let me explain what I mean by noise. I mean, just randomness. And it’s true within individuals, but it’s especially true among individuals who are supposed to be interchangeable in, say, organizations. ...

I’ll tell you where the experiment from which my current fascination with noise arose. I was working with an insurance company, and we did a very standard experiment. They constructed cases, very routine, standard cases. Expensive cases — we’re not talking of insuring cars. We’re talking of insuring financial firms for risk of fraud.

So you have people who are specialists in this. This is what they do. Cases were constructed completely realistically, the kind of thing that people encounter every day. You have 50 people reading a case and putting a dollar value on it.

I could ask you, and I asked the executives in the firm, and it’s a number that just about everybody agrees. Suppose you take two people at random, two underwriters at random. You average the premium they set, you take the difference between them, and you divide the difference by the average.

By what percentage do people differ? Well, would you expect people to differ? And there is a common answer that you find, when I just talk to people and ask them, or the executives had the same answer. It’s somewhere around 10 percent. That’s what people expect to see in a well-run firm.

Now, what we found was 50 percent, 5–0, which, by the way, means that those underwriters were absolutely wasting their time, in the sense of assessing risk. So that’s noise, and you find variability across individuals, which is not supposed to exist.
And you find variability within individuals, depending morning, afternoon, hot, cold. A lot of things influence the way that people make judgments: whether they are full, or whether they’ve had lunch or haven’t had lunch affects the judges, and things like that.
Now, it’s hard to say what there is more of, noise or bias. But one thing is very certain — that bias has been overestimated at the expense of noise. Virtually all the literature and a lot of public conversation is about biases. But in fact, noise is, I think, extremely important, very prevalent.

There is an interesting fact — that noise and bias are independent sources of error, so that reducing either of them improves overall accuracy. There is room for . . . and the procedures by which you would reduce bias and reduce noise are not the same. So that’s what I’m fascinated by these days.

Now, it’s hard to say what there is more of, noise or bias. But one thing is very certain — that bias has been overestimated at the expense of noise. Virtually all the literature and a lot of public conversation is about biases. But in fact, noise is, I think, extremely important, very prevalent. ...

COWEN: Do you see the wisdom of crowds as a way of addressing noise in business firms? So you take all the auditors, and you somehow construct a weighted average? ...

KAHNEMAN: With respect to the underwriters, I would expect, certainly, that if you took 12 underwriters assessing the same risk, you would eliminate the noise. You would be left with bias, but you would eliminate one source of error, and the question is just price. Google, for example, when it hires people, they have a minimum of four individuals making independent assessments of each candidate. And that reduces the standard deviation of error at least by a factor of two.

COWEN: So is the business world, in general, adjusting for noise right now? Or only some highly successful firms?

KAHNEMAN: I don’t know enough about that. All I do know is that, when we pointed out the results, the bewildering results of the experiment on underwriters, and there was another unit — people who assess the size of claims. Again, actually, it’s more than 50 percent. Like 58 percent. The thing that was the most striking was that nobody in the organization had any idea that this was going on. It took people completely by surprise.
My guess now, that wherever people exercise judgment, there is noise. And, as a first rule, there is more noise than people expect, and there’s more noise than they can imagine because it’s very difficult to imagine that people have a very different opinion from yours when your opinion is right, which it is. ...

COWEN: If you’re called in by a CEO to give advice — and I think sometimes you are — how can I reduce the noise in my decisions, the decisions of the CEO, when there’s not a simple way to average? The firm doesn’t have a dozen CEOs. What’s your advice? ...
KAHNEMAN: [T]here is one thing that we know that improves the quality of judgment, I think. And this is to delay intuition. ... Delaying intuition until the facts are in, at hand, and looking at dimensions of the problem separately and independently is a better use of information.

The problem with intuition is that it forms very quickly, so that you need to have special procedures in place to control it except in those rare cases ...  where you have intuitive expertise. That’s true for athletes — they respond intuitively. It’s true for chess masters. It’s true for firefighters ... I don’t think CEOs encounter many problems where they have intuitive expertise. They haven’t had the opportunity to acquire it, so they better slow down. ... It’s not so much a matter of time because you don’t want people to get paralyzed by analysis. But it’s a matter of planning how you’re going to make the decision, and making it in stages, and not acting without an intuitive certainty that you are doing the right thing. But just delay it until all the information is available.

COWEN: And does noise play any useful roles, either in businesses or in broader society? Or is it just a cost we would like to minimize?

KAHNEMAN: There is one condition under which noise is very useful. If there is a selection process, evolution works on noise. You have random variation and then selection. But when there is no selection, noise is just a cost. ... Bias and noise do not cover the universe. There are other categories.

A version of this article first appeared on Conversable Economist

Share this article

Leave your comments

Post comment as a guest

terms and condition.
  • Alex Paige

    Such a smart guy !!!

  • Zack Parsley

    This man is brilliant!

  • John Wilson

    Thank you for your work Dnaiel Kahneman.

  • Ethan Brown

    Sounds complicated

  • Andy Doyle

    The last sentence makes a lot of sense to me

Share this article

Timothy Taylor

Global Economy Expert

Timothy Taylor is an American economist. He is managing editor of the Journal of Economic Perspectives, a quarterly academic journal produced at Macalester College and published by the American Economic Association. Taylor received his Bachelor of Arts degree from Haverford College and a master's degree in economics from Stanford University. At Stanford, he was winner of the award for excellent teaching in a large class (more than 30 students) given by the Associated Students of Stanford University. At Minnesota, he was named a Distinguished Lecturer by the Department of Economics and voted Teacher of the Year by the master's degree students at the Hubert H. Humphrey Institute of Public Affairs. Taylor has been a guest speaker for groups of teachers of high school economics, visiting diplomats from eastern Europe, talk-radio shows, and community groups. From 1989 to 1997, Professor Taylor wrote an economics opinion column for the San Jose Mercury-News. He has published multiple lectures on economics through The Teaching Company. With Rudolph Penner and Isabel Sawhill, he is co-author of Updating America's Social Contract (2000), whose first chapter provided an early radical centrist perspective, "An Agenda for the Radical Middle". Taylor is also the author of The Instant Economist: Everything You Need to Know About How the Economy Works, published by the Penguin Group in 2012. The fourth edition of Taylor's Principles of Economics textbook was published by Textbook Media in 2017.


Latest Articles

View all
  • Science
  • Technology
  • Companies
  • Environment
  • Global Economy
  • Finance
  • Politics
  • Society
Cookies user prefences
We use cookies to ensure you to get the best experience on our website. If you decline the use of cookies, this website may not function as expected.
Accept all
Decline all
Read more
Tools used to analyze the data to measure the effectiveness of a website and to understand how it works.
Google Analytics