Loading...
You are here:  Home  >  Data Education  >  BI / Data Science News, Articles, & Education  >  BI / Data Science Blogs  >  Current Article

Politics, Luck, Bayes Theorem, and Base Rates

By   /  October 4, 2017  /  No Comments

Click to learn more about author Steve Miller.

My wife gave me a stern warning before we headed out to a suburban neighbor’s Super Bowl party early this year: avoid “spirited” political discussion or else. Actually, I’d already self-forewarned, the memory of  being in the center of a blue vs red imbroglio several years ago fresh in mind. And it’s safe to say there was heightened sensitivity this year.

After a pretty benign gathering for the first 90 minutes or so, I did get a bit animated when the discussion turned to the roles of skill, hard work, and good fortune with career success, especially since I ‘d just finished an interesting little read, “Success And Luck: Good Fortune and the Myth of Meritocracy”, by Robert Frank.

A blue economist, Frank forwards  “How important is luck in economic success? No question more reliably divides conservatives from liberals. As conservatives correctly observe, people who amass great fortunes are almost always talented and hardworking. But liberals are also correct to note that countless others have those same qualities yet never earn much. In recent years, social scientists have discovered that chance plays a much larger role in important life outcomes than most people imagine. In Success and Luck, bestselling author and New York Times economics columnist Robert Frank explores the surprising implications of those findings to show why the rich underestimate the importance of luck in success—and why that hurts everyone, even the wealthy.”

Frank devotes much of his book to offering evidence that successful people are also lucky and that false beliefs on the attribution of luck and skill actually promote success. In a variant of the privatize the gains, socialize the losses bromide, successful people often attribute their losses to bad luck and their wins to skill.

From the blue side,  I challenged the “luck’s a non-factor – anyone can become rich, just like me” position of my red investment adviser neighbor. He pooh-poohed the suggestion that the stock market performance under the previous blue administration might have been a contributor to his run of financial success, crowing “I make money in good economies and bad”. I’m sure he’s right, but bet he did better the last eight years than the eight before.

I decided not to offer Frank’s evidence from psychology to the discussion, though. My counter was a more mathematical one, translating my neighbor’s “work hard get rich” argument into the statistical logic of Reverend Thomas Bayes.

Bayes Theorem has to do with conditional probability – the likelihood of event B happening, given that event A has occurred , designated as P(B/A). Bayes Theorem is currently of  increasing importance in  probability and statistics, narrowing the gap between mathematical models and experience/beliefs in the statistical world. At the same time, Bayes is also of keen interest in cognitive science, because it engages a human/psychological element that’s quite fallible. Indeed, in analytics, Bayes is lauded for bridging the modeling/experience divide, while simultaneously derided for its lack of precision with experience/existing knowledge.

In algebraic terms, Bayes Theorem states that P(B/A) = P(A/B)*P(B)/P(A). The left side term,  P(B/A), is called the posterior probability, while P(B) is designated the prior probability or base rate. P(A/B) is known as the likelihood.

In our career success parlance, the translations would be as follows: the posterior probability that one gets rich, given that she has worked hard, is designated as P($/W). The base rate or prior, P($), is the unconditional probability that one is rich. The likelihood, P(W/$), is the probability that one works hard, given that she’s rich. Finally, the unconditional probability that one works hard is P(W).

Let’s be generous and say the probability of getting rich, P($), is .1, while the probability of working hard, given rich, P(W/$), is high – maybe .9. Finally, let’s assume that the unconditional probability of working hard, P(W), is .8. Using Bayes Theorem, we then compute the probability of getting rich, given hard work, P($/W), as .9*.1/.8 = .1125 or 11.25% – anything but a slam dunk.

The main culprit for the low conditional probability is the .1 base rate (or prior), P($). Nobel prize-winning cognitive scientist Daniel Kahneman notes that people often make such “statistical base rate underweighting” errors when calculating “conditional probabilities” in their day to day thinking.

My “anyone can get rich like him through hard work” neighbor’s reaction to the surprising Bayes results? Much like subjects in base rate experiments, he responds more to the story or individuating information than to the accompanying statistical support. In short, he didn’t buy it.

Can’t wait till next year and Super Bowl LII!

About the author

Steve Miller, Co-founder and President of Inqudia Consulting A Co-founder and current President of Inquidia Consulting, Steve Miller has over 35 years experience in business intelligence and statistics, the last 25 revolving on the delivery of analytics technology services. Prior to Inquidia, Steve held positions as Executive Vice President for Piocon Technologies, Executive Vice President for Braun Consulting, and Sr. Principal for Oracle Consulting. He studied Quantitative Methods and Statistics at Johns Hopkins University, the University of Wisconsin, and the University of Illinois.

You might also like...

To Get Value from Data, Organizations Should Also Focus on Data Flow

Read More →