A Nobel effort

This year's Nobel Memorial Prize in Economic Sciences has been awarded to three path-breaking economists for their empirical analysis of asset prices.

Eugene Fama is the father of the Efficient Market Hypothesis (EMH) which states that financial markets instantly factor in all known information into asset prices. One implication of this hypothesis is that professional investors cannot outperform the market except by luck. Therefore the best strategy is to simply invest in an index fund.

Robert Shiller challenged the EMH with his findings that the volatility of stock prices is greater than would be expected by the changes in dividends. His work has been part of the behavioral revolution in economics which studies the psychological factors in economic decisions and their effect on markets.

Lars Peter Hansen developed statistical tools that greatly simplified the analysis of asset prices.

Of relevance to all this, in Thinking Fast and Slow, Daniel Kahnemen describes his statistical research that demonstrates the inability of professional investors (including fund managers) to consistently beat the market. As he puts it, "There is general agreement among researchers that nearly all stock pickers, whether they know it or not - and few of them do - are playing a game of chance. The subjective experience of traders is that they are making sensible educated guesses in a situation of great uncertainty. In highly efficient markets, however, educated guesses are no more accurate than blind guesses."

Plausible denial

Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.

Which is more probable?
  1. Linda is a bank teller.
  2. Linda is a bank teller and is active in the feminist movement.

Also think about why you think the option you chose is more probable.

Cognitive psychologists Tversky and Kahneman posed this question in an experimental study and found that 90% of respondents chose the second option. However the correct answer is the first option. The reason it is correct is that two conditions occurring (being a bank teller and being a feminist) is less likely than for one of the conditions to occur (being a bank teller regardless of whether she is a feminist or not). Tversky and Kahneman coined this the Conjunction Fallacy.

What is true is that the second option is a more representative, more coherent and more plausible match with Linda's personality description. Linda doesn't really seem the bank teller type but being a feminist would certainly be in character. Thus, instead of making a judgment about probability, most people substitute a judgment about similarity instead.

Does this actually matter in practice? Yes, it matters when important life choices are made based on similarities rather than probabilities. For example, a doctor may diagnose a patient's illness due to symptoms being representative of a particular disease (or even of having no disease at all) when in fact other diseases are more likely.

Conditioned thinking

Steve is a shy and tidy man. Is Steve more likely to be a librarian or a farmer?

 

With a few facts about librarians and farmers, it is possible to calculate the exact likelihood. In the US, there are twenty times more male farmers than male librarians. If four out of five librarians are shy and tidy and only one in five farmers have both those personality traits, then how likely is it that Steve is a librarian rather than a farmer?

 

Bayes' Theorem enables us to calculate the precise figure. The answer, perhaps surprisingly, is that Steve has only a seventeen percent chance of being a librarian rather than a farmer.

 

How can this be? If you guessed that Steve was more likely to be a librarian than a farmer then why does this intuitive answer contradict the calculated answer? The reason is that while we readily recognize the stereotypical personalities of librarians and farmers, we tend to ignore the general population sizes of the groups we are analyzing. The population sizes complicate the calculation, so we use a representative heuristic instead that neglects the relevant base rates.

 

The moral of the story is that we should try to identify base rate information and allow it to moderate our intuitive conclusions. In this case, the knowledge that there are many more male farmers than male librarians means that the answer is likely to be much less than Steve's resemblance to librarians suggests.

 

--

 

I came across the librarian story in Thinking, Fast and Slow by Daniel Kahneman. Working through the problem, we have:

 

  L = Librarian, S = Shy and tidy, P(L|S) is the probability of L given S

  P(L) = 1/20 [ the probability of being a librarian (actually 1:20 which is 1/21, but rounded for simplicity) ]
  P(S|L) = 4/5 [ the probability that a male librarian is shy and tidy ]
  P(S|~L) = 1/5 [ the probability that a male farmer is shy and tidy ]
  P(S) = P(S|L) * P(L) + P(S|~L) * P(~L) = 4/5 * 1/20 + 1/5 * 19/20 = 23/100 [ the probability of being shy and tidy ]

  Now plug the above values into Bayes' Theorem:
  P(L|S) = P(S|L) * P(L) / P(S) = 4/5 * 1/20 / 23/100 = 4/23 (17%) [ the probability that a shy and tidy male is a librarian ]