Nate Silver’s “The Signal and the Noise”: Book Review, Notes + Analysis

Poor Ash’s Almanack > Book Reviews > Science + Engineering + Math > statistics/forecasting

Overall Rating: ★★★★★★ (6/7) (standout for its category)

Per-Hour Learning Potential / Utility: ★★★★★★ (6/7)

Readability: ★★★★★★ (6/7)

Challenge Level: 3/5 (Intermediate) | 450 pages ex-notes (560 official)

Blurb/Description:  FiveThirtyEight founder Nate Silver provides a thorough yet digestible/approachable analysis of predictions and forecasting: what we do wrong, what we do well, and how we can improve.

Summary: This is an excellently-written, conversational book that is far more engaging and thought-provoking than I thought it would be, delving into fields ranging from meteorology to sports betting to poker to, of course, economics and finance.  I mean, just take a look at this:

Ice cream sales and forest fires are correlated because both occur more often in the summer heat. But there is no causation; you don’t light a patch of the Montana brush on fire when you buy a pint of Haagen-Dazs. - Nate Silver Click To Tweet

Most of all, it’s very applicable – on my first read through, I found myself stopping every so often to think about how I could incorporate bits of it into my process.

Highlights:  For someone who spends all day thinking about numbers, Silver (pictured at right) is surprisingly great with vivid metaphors, such as this the above-quoted one about correlation vs. causation, and the “cheating spouses” exploration of Bayesian reasoning.

His storytelling is also great – maybe my favorite part of the book is the story behind the eponymous chapter “FOR YEARS YOU’VE BEEN TELLING US THAT RAIN IS GREEN.”

The book is also thoughtful and practical in all places.  While it’s somewhat narrower in scope than, say, Jordan Ellenberg’s How Not To Be Wrong (HNW review + notes), I think that focus is appropriate, because Silver deeply explores an important topic.

Lowlights: In some places, Silver’s book is overly repetitive and/or detailed; on my second read-through, I found myself skimming and getting more or less just as much out of it (although of course that was with the benefit of a word-for-word reading the first time).

Mental Model / ART Thinking Points:  storytellingoverconfidenceBayesian reasoningcorrelationvs. causationnonlinearityproduct vs. packagingfeedbacksample sizebase ratesmargin of safetyutilityprecision vs. accuracy, complexity,

You should buy a copy of The Signal and The Noise if: you’re interested in improving your forecasts.

Reading Tips: Skip chapter 11; it’s not very good.

Pairs Well With:

The Halo Effect by Phil Rosenzweig (Halo review + notes) – not as focused on data analysis, but also gets into the issue of “signal vs. noise.”  A must-read for those who read business books.

How Not To Be Wrong by Jordan Ellenberg (HNW review + notes) – while I prefer Silver’s The Signal and the NoiseHow Not To Be Wrong covers a lot of important mental models as well, and is generally very well-written.

The Success Equation by Michael Mauboussin ( TSE review + notes).  A great and concise exploration of skill and luck.

Superforecasting” by Philip Tetlock (SF review + notes).  A book about predictions in another context: how can ordinary individuals like you and I outperform experts in their own fields?  The answer turns out to be a specific thought process.

Reread Value: 4/5 (High)

More Detailed Notes + Analysis (SPOILERS BELOW):

IMPORTANT: the below commentary DOES NOT SUBSTITUTE for READING THE BOOK.  Full stop. This commentary is NOT a comprehensive summary of the lessons of the book, or intended to be comprehensive.  It was primarily created for my own personal reference.

Much of the below will be utterly incomprehensible if you have not read the book, or if you do not have the book on hand to reference.  Even if it was comprehensive, you would be depriving yourself of the vast majority of the learning opportunity by only reading the “Cliff Notes.”  Do so at your own peril.

I provide these notes and analysis for five use cases.  First, they may help you decide which books you should put on your shelf, based on a quick review of some of the ideas discussed.  

Second, as I discuss in the memory mental model, time-delayed re-encoding strengthens memory, and notes can also serve as a “cue” to enhance recall.  However, taking notes is a time consuming process that many busy students and professionals opt out of, so hopefully these notes can serve as a starting point to which you can append your own thoughts, marginalia, insights, etc.

Third, perhaps most importantly of all, I contextualize authors’ points with points from other books that either serve to strengthen, or weaken, the arguments made.  I also point out how specific examples tie in to specific mental models, which you are encouraged to read, thereby enriching your understanding and accelerating your learning.  Combining two and three, I recommend that you read these notes while the book’s still fresh in your mind – after a few days, perhaps.

Fourth, they will hopefully serve as a “discovery mechanism” for further related reading.

Fifth and finally, they will hopefully serve as an index for you to return to at a future point in time, to identify sections of the book worth rereading to help you better address current challenges and opportunities in your life – or to reinterpret and reimagine elements of the book in a light you didn’t see previously because you weren’t familiar with all the other models or books discussed in the third use case.

“This is a book about prediction – a study of why some predictions succeed and why some fail,” starts off Nate Silver, essentially saying: if you’re a human being trying to make decisions, you need to read this book.  Wittily, on overconfidence:

We love to predict things, and we aren’t very good at it. - Nate Silver Click To Tweet

Philip Tetlock would agree – Superforecasting (SF review + notes) provides a great exploration of why many expert forecasts fail, and how ordinary people can use a specific mental process to do better.

Silver starts off by exploring how iInformation has exploded since the invention of the printing press, which reduced the cost of a book from $20K to $70 (in circa-201X USD) and made it much more accurate, given that previous books were manual copies of copies of copies.

Here is where you think back to playing the telephone game at summer camp – Silver notes the example of a Bible which proclaimed thou SHALT commit adultery,” capitals mine.

Unfortunately, more information does not always equate to better decisions – precision vs. accuracy – our response to information overload is selective engagement, i.e. confirmation bias selective perception, and so on.

Obviously, in the era of Big Data, it looks like this problem is getting harder rather than easier, and Silver’s firmly not in the techno-utopia camp: his thesis is that:

“Numbers have no way of speaking for themselves.  We speak for them. We imbue them with meaning. […] It is when we deny our role in the process that the odds of failure rise.  Before we demand more of our data, we need to demand more of ourselves.

This is essentially Silver’s defense of Bayesian reasoning (which we’ll get to).  Jordan Ellenberg makes similar points in parts of How Not To Be Wrong (HNW review + notes).  To wit, from pages 178 – 180 of HNW:

“You’d like to say that your beliefs are based on evidence alone, not on some prior preconceptions you walked in the door with.  

But let’s face it – no one actually forms their beliefs this way. If an experiment […] slowed the growth of […] cancer […] by putting patients inside ay plastic replica of Stonehenge, would you grudgingly accept that [vibrational earth energy was curative?]  

You would not, because that’s nutty. You’d think Stonehenge probably got lucky.

You have different priors about those two theories, and as a result you interpret the evidence differently, despite it being numerically the same.”

Back to Silver: he notes that you might be surprised by his take, given his background as a stathead – and I certainly was.  I put off reading this book for a while because statistics was almost bar-none my least favorite class in B-school (tie between that and organizational behavior, which I felt like sort of boiled down to “please don’t sexually harass your employees” – I guess that wasn’t a prereq at Uber and so on.)  Tangents aside, if you think Silver is the computer nerd coming to take your humanity from you, have no fear: that’s not what this book is about.

Silver brings up the premise of biology: we’re built with powerful heuristics that allow us to recognize patterns efficiently (recall how astonishingly long it took for computers to be able to efficiently recognize a cat, something which most human toddlers can do with ease.)  

That example is explored in Peter Thiel’s “ Zero to One ( Z21 review + notes):

“men and machines are good at fundamentally different things.  People have intentionality – we form plans and make decisions in complicated situations.  We’re less good at making sense of enormous amounts of data. Computers are exactly the opposite: they excel at efficient data processing, but they struggle to make basic judgments that would be simple for any human.  […]

in 2012, [Google’s supercomputer] learned to identify a cat with 75% accuracy […] remember that an average four-year-old can do it flawlessly.”  

Unfortunately, our pattern-recognition capabilities make us (hu)man(s)-with-a-hammer – as MIT neuroscientist Tomaso Poggio puts it succinctly, we find patterns in random noise all the time.

As much information as we can process, it amounts to merely basis points of basis points of the information that’s being produced every day (let alone out there in total), so we have to be selective about what we process.

Silver disputes the notion that there is no objective truth, and the notion that a hypothesis is not scientific unless it is falsifiable – because not all predictions are cleanly testable in the new world.  What Silver proposes in its stead is a Bayesian probabilistic approach to the world.

Silver’s first major topic of investigation is the financial crisis: AAA-rated CDOs failed at 200 times the rate predicted; 28% vs. 0.12%.  He turns to the ratings agencies, which seemed to have sufficient information to make a better prediction (or at least update their prediction earlier – and not text you to let you know your house is on fire when you’re already filing the insurance paperwork.)

One of the major problems was that the models assumed that the underlying assets weren’t correlated – which they weren’t, in good times, but in bad times it turns out they were.  One of the major correlating factors: the significant rise in home prices nationwide.

While Moody’s did try to adjust, it wasn’t good enough.  Silver on  nonlinearity interacting with margin of safety:

“[Moody’s 50% adjustment] might have been fine had the potential for error in their forecasts been linear and arithmetic.  

But leverage, or investments financed by debt, can make the error in a forecast compound many times over, and introduces the potential of highly geometric and nonlinear mistakes.  

Moody’s 50 percent adjustment was like applying sunscreen and claiming it protected you from a nuclear meltdown.

Silver offers different definitions of risk and uncertainty: you can put a price on risk, but you have no idea on uncertainty.  The ratings agencies tried to alchemize uncertainty into risk.

For emphasis:

Leverage can make the error in a forecast compound many times over, and introduce… nonlinear mistakes. Moody’s 50 percent adjustment was like applying sunscreen and claiming it protected you from a nuclear meltdown. – Nate Silver Click To Tweet

Housing bubbles were not entirely unknown – they had precedent in areas like Norway and Japan, and Silver makes the case that the data for U.S. home prices were very anomalous relative to long-term history.  Even once things started to hit the fan, expert predictions were way off: in December 2007 (when the U.S. was already in recession), a WSJ economist panel predicted only a 38% chance of recession within the next year; another panel thought the chance of the outcome that actually happened was less than 1-in-500.

Interesting read-across to quality businesses in Silver’s discussion of the tendency of information-asymmetry markets to put out a worse and worse product over time – “there may be no such thing as a fair price” when buying an asset from a stranger who knows much more than you do and won’t let you take a test-drive – quality as a stand-in for test-driving in investing?

Silver brings up the concept of feedback loops as a contributor – the more people bought MBS, the more they were rated and the more people felt safe buying them, etc.  Bubbles, like  stress, can be an autocatalytic process where more begets more, as I explore in the feedback mental model.

The commonality behind the failed predictions is that the situation was out-of-sample: the predictions were being made upon data that had no validity – Silver’s example is that if you’re a good driver but have never driven drunk, it doesn’t matter what your previous driving record was if you get behind the wheel after twelve vodka tonics.    Sample size.

(Public service announcement: as Dr. Matthew West explores in some depth in “Why We Sleep” – Sleep review + notes – more people are killed by drowsy drivers than by drunk drivers; even modest sleep deprivation can impair us to the same extent as being legally drunk.)

Back to Silver: Moody’s data was based on a sample of flat to rising house prices… so the model wasn’t prepared to handle what actually happened.

One of the key takeaways from this chapter is that better predictions could have been made… but a powerful force against them was behavioral – Silver notes on  incentives and product vs. packaging:

“forecasters often resist considering out-of-sample problems… the model will seem to be less powerful, look less impressive.  Our personal and professional incentives almost always discourage us from doing this.”  

Recall Thaler’s Misbehaving (M review + notes).  At one of Thaler’s presentations on behavioral economics, one unusually candid economist literally asked Thaler: if your newfangled theory is correct, what do I do?  I’ve spent my entire career figuring out how to do it the old way…

As the saying goes, all models are wrong, but some models are useful – clearly, though, models built for impressing others rather than the truth aren’t useful

Silver makes this point too, to wit, on precision vs. accuracy:

this syndrome is often associated with very precise-seeming predictions that are not at all accurate.  Moody’s carried out their calculations to the second decimal place – but they were utterly divorced from reality.  

This is like claiming you are a good shot because your bullets always end up in about the same place – even though they are nowhere near the target.”

In How Not To Be Wrong (HNW review + notes), Ellenberg makes a similar point, going back to his missile riff: it doesn’t matter if the missile will hit your house in 4.3 or 4.5 seconds… you better get outta Dodge.

The next section is a bit less quotable, going into some detail about the (in)accuracy of forecasts by television pundits, including election forecasts.  Silver references the Hedgehog-vs-Fox dichotomy, which was also covered (with a different perspective) in Rosenzweig’s  The Halo Effect (Halo review + notes) as well as Philip Tetlock’s “ Superforecasting(SF review + notes).

Ultimately, he believes in acknowledging real-world uncertainty, avoiding commitment bias (constantly updating), and trying to ask a lot of questions and search for alternate types of information.  Additionally, and perhaps most interestingly, he believes in integrating qualitative with quantitative analysis – he believes it’s not so much which that’s important, but rather, how you weight it…

Silver continues with a discussion of – you guessed it – baseball, which is unique among sports for its ability to use statistics (full disclosure – I’ve never ready Moneyball, nor do I particularly care about baseball, but I still liked this section).  The interesting conclusion is, again, that integrating statistics with human judgment seems to work best.

The next chapter is quite interesting, covering weather forecasting (a success story) with detours into philosophy, chaos theory / complexity, and other areas – Silver is nothing if notmultidisciplinary, which I admire – he even throws in the occasional obligatory joke about weather forecasts.

Silver quips that the Weather Service was organized in 1870 as a subsidiary of the Department of War:

partly because the whole enterprise was so hopeless that it was only worth bothering with during wartime when you would try almost anything to get an edge.”  

Jokes aside, though, the takeaway here is that humans can also add an edge with good judgment – even as computers get more powerful, humans have fairly consistently improved the accuracy of precipitation forecasts by 25% and temperature forecasts by 10%, per government data.

Going back to the concept of all models being wrong vs. some models being useful, one of the interesting takeaways from this chapter is that the most accurate forecasts may not be the best or most useful – in the sense that, the costs of not expecting precipitation and getting it (not carrying an umbrella, or planning an outdoor excursion) usually exceed the costs of expecting precipitation and not getting it (i.e. carrying an umbrella, or postponing the excursion to another day).

This is sort of margin of safety applied to weather, in other words. That’s my takeaway, anyway – Silver disagrees, stating that:

“it is forecasting’s original sin to put politics, personal glory, or economic benefit before the truth of a forecast”

But (with apologies to Damodaran, who takes a different tack,) I’d rather be conservative and surprised to the upside than try to be precisely correct and surprised to the downside, and I think this is the correct approach for most business managers and investors.

The next chapter is about earthquakes, which, unlike hurricanes, cannot be predicted (in a time-dependent way) with much accuracy.  Pages 163 – 171 provide a very good discussion of backtest overfitting (also see this paper), which is a problem with a lot of quantitative strategies – Silver notes that when data is very noisy, and the analyst “doesn’t know or doesn’t care about the truth of the relationship,” overfitting is a risk.  

(This would appear to be one of the central underpinnings of the idea between mixing qualitative and quantitative analysis – while our intuitive insights may often be wrong, so too are correlations, like the price of butter in Bangladesh and the S&P 500 or whatever, so having some idea of why there might be causality would seem to strengthen the model.)

This relates to priors and  base rates – in How Doctors Think (HDT review + notes), Dr. Jerome Groopman cites one doctor who:

“emphasizes to his interns and residents in the ER that they should not order a test unless they know how that test performs in a patient with the condition they assume he has.  

That way, they can properly weight the result in their assessment.”

Groopman is, however, generally skeptical on Bayesian reasoning; it’s an interesting perspective.

The next chapter, titled “How to Drown in Three Feet of Water,” is very thought-provoking.  It covers the concept of margin of error – focusing initially on economists’ forecasts, wherein the 90% confidence interval was broken between a third and half of the time, per various studies.  

Silver also brings up the issue I just mentioned – i.e. correlation vs.causation – noting that the “Super Bowl Indicator”  (NFL vs. AFL predicting the stock market) would have had a 1-in-4.7-million probability of being statistically insignificant – but of course, it’s bogus.  

It’s easy to see why this is nonsense, but what’s harder is when you have tons of economic data – just by virtue of there being millions of things you could analyze, you will eventually find some great indicator if you look hard enough – but that says nothing about causality.  Silver comes back to this point at the end of the chapter after talking more about economic forecasts, noting what he sees as the flaws in the “Big Data” mentality of just listening to the numbers – when the numbers are noisy, they may mislead you.

Silver moves on to provide some interesting discussion of historical projections of the flu and population growth (the takeaway on the latter being that with a long-enough time horizon, small errors in exponential forecasts go horribly wrong).  Self-fulfilling and self-limiting can also be an important phenomenon: for example, the diagnosis rate of autism seems to track the media coverage of it.  See  feedback.

Silver provides another example of the success of mixing qualitative with quantitative data to forecast things (sports betting), then introduces Bayesian reasoning as a counterpoint to the standard, dogmatic “frequentist” approach to statistical analysis (the implicit assumption of which is that with enough data, your error goes to zero.)  This is one of the best parts of the book; he also here cites the Ioannidis paper – see my notes on Jordan Ellenberg’s “ How Not To Be Wrong ( HNW review + notes) for more discussion of that.

Silver proceeds to talk about chess and Deep Blue, and how Kasparov managed to beat the computer by taking it out of the database (see also The Most Human Human).  This isn’t really Silver’s point here per se, but I really like this line about mindfulness and  intellectual humility:

Elite chess players tend to be good at metacognition – thinking about the way they think – and correcting themselves if they don’t seem to be striking the right balance.

See also Tetlock on page 236 of “ Superforecasting ( SF review + notes);

“My sense is that some superforecasters are so well practiced in System 2 corrections – such as stepping back to take the outside view – that these techniques have become habitual.  In effect, they are now part of their System 1.  

Silver’s next chapter is about poker, which is interesting but doesn’t have a ton of takeaways – other than play easy gameskeep your emotions out of it, and differentiate between luck and skill (see also Mauboussin’s “ The Success Equation”  – TSE review + notes).

Chapter 11, about the stock market, isn’t very interesting or good or useful (although this may be because I, and most of us, are already very familiar with this material.)

Chapter 12 was an interesting and (in my view) measured and thoughtful analysis of climate science, but no huge takeaways; the same goes for the next chapter about terrorism.

The book concludes with a useful summary.

First Read: 2015

Last Read: 2017

Number of Times Read: 2

Planning to Read Again?: yes

 

Review Date: summer 2017

Notes Date: summer 2017