The Flaw of Averages by Sam L. Savage

Date finished 27 Feb 2021

A book on assessing risk in the face of uncertainty, focussing as the title suggests on how not to assess it. The author is quirky and he makes up for the dryness of the subject by packing the chapters with examples and analogies; he’s funny (occasionally). I didn’t find anything drastically new but it was a good refresher; I took lots of notes.

Notes

"”The only certainty is that nothing is certain”.

• Foundations

  • The Flaw of averages: Plans based on average assumptions are wrong on average.
    • The statistician who drowned whilst fording a river that was, on average, only three feet deep. (cartoon by Jeff Danziger)
  • Towards the start of the book gives the example of people leaving work to get home, how often will we get home in time to leave by 6pm. That with even 1+ more people (dependencies) the chances rapidly decrease.
  • The phrase “Give me a number” is a dependable leading indicator of an encounter with the Flaw of Averages. You need a range!
  • Patrick Leach in a book called Why Can’t You Just Give Me a Number? points out that “once a value is generated, put down on paper, and incorporated into the business plan, it becomes gospel.”
  • Regression analysis with single data points are bound to mislead. Example, future growth of housing values.
  • Monte Carlo Simulation
    • The thing you do before climbing a ladder to paint your house is to give it a good shake. By bombarding it with random physical forces, you simulate how stable the ladder will be when you climb it. You can then adjust accordingly so as to minimise the risk that it falls down with you on it.
    • Test stability of uncertain business plans etc.
    • Input probability distribution - the shaking forces applied to the ladder.
    • Output probability distribution - the movements of the ladder.
    • Developed as part of the Manhattan Project.
    • Simulation does for uncertainty what a lightbulb does for darkness.
    • Most managers have no idea how to generate Input probability distribution.
      • scenario libraries exist - repositories on corporate intelligence on uncertainty.
    • You cannot add the results of simulations - you must simulate the whole thing together (eg. 2 ladders joined). You need collaboration internally in organisations to do this!
  • Models
    • The Wright Brothers’ first model was a bicycle inner tube box.
      • Models were key to their success.
    • The most important models are like embryos that may not resemble the final product but nonetheless contain the developmental necessities of the application (DNA).
    • “All Models are wrong, some models are useful” - George Box
      • e.g. Isaac Newton’s models of physical motion were superseded by relativistic models of Einstein. But Newton built the foundations from which Einstein started.
    • “You are allowed to lie a little, but you must never mislead” - Paul Halmos
      • Models do not represent the entire truth, but they should lead to the truth.
    • “A successful model tells you things you didn’t tell it to tell you”. - Jerry P. Brashear
      • model the things you don’t understand rather than that which you do.
    • Knuth’s 5 stages of model development
      1. Decide what you want the model to do.
      2. Decide how to build the model.
      3. Build the model.
      4. Debug the model.
      5. Trash stages 1 through 4 and start again, now that you know what you really wanted in the first place.
    • To get a large model to work, you must start with a small model that works, not a large model that doesn’t work.
    • “Far better an approximate answer to the right question, which is often vague, than the exact answer to the wrong question, which can always be made precise.” - John W. Tukey
  • “The single biggest problem in communication is the illusion that it has taken place” - George Bernard Shaw

• Concepts (mindles are ‘handles’ for uncertainty)

  • Mindle1: Uncertainty vs risk
    • Risk is in the eye of the beholder. It’s relative.
    • Risk reflects how uncertain outcomes cause loss or injury to a particular individual or group.
    • Risk attitude (utility theory) is the willingness to incur risk in the quest of reward.
  • Mindle 2: An uncertain number is a shape
    • Random variable = uncertain number … a shape - a distribution.
    • A histogram is a nice way to view probability distribution.
    • Bosses should be asking for distributions rather than numbers.
    • The cumulative distribution shows the probability that the number is less than a given value.
    • The average also known as the mean, or expected value, of the uncertain number is the balance point of the distribution.
    • Black Swans; events that 1) never had occurred before, 2) would have an extreme impact if it did occur and 3) is easy to explain after the fact are obviously not included in the distribution.
    • The median is the quantity that the uncertain number has a 50/50 chance of being greater than or less than. Not always the same as the average… Warren Buffet joining a room effect on average wealth example.
    • The mode is the place at which the histogram has its highest peak.
  • Mindle 3: Combinations of uncertain numbers
    • A combination of uncertain numbers is a shape that goes up in the middle.
    • This effect arises from diversification.
    • When enough independent uncertain numbers are added together, the resulting distribution becomes bell-shaped.
    • Diversification reduces the chance of extreme outcomes. Dice example, Spinners, investment example.
    • central limit theorem = diversification.
    • normal distribution = bell-shaped distribution.
    • Use:
      • Histogram
      • Cumulative graph
      • Percentiles
      • Variation
    • Don’t use:
      • Sigma or standard deviation or variance measure how wide the distribution of an uncertain number is. Look at its shape, the wider the distribution the greater the possible variation and the higher the Sigma and the variance.
  • Mindle 4: Terri Dial and the Drunk in the Road
    • “Consider a drunk staggering down the middle of a busy highway and assume that his average position is the centreline. Then the state of the drunk at his average position is alive, but on average he’s dead”.
    • The strong form of the flaw of averages states that average or expected inputs don’t always result in average or expected outputs.
      • You can limit downside to making a loss by simply shutting.
  • Mindle 5: Interrelated uncertainties
    • Knee-jerk reaction is to reject petroleum and airlines because the elements are interrelated. The correct knee-jerk reaction is to reject portfolios whose interrelations are direct (or positive). Eg two airline stocks will be positively correlated, they will likely tank together. Whereas when petroleum goes down it’s good for airlines.
    • Inverse or negative interactions reduce uncertainty, and the name for an investment added to a portfolio for this purpose is a hedge.
    • Scatter plots are the best way to grasp the interrelationships between uncertain numbers.
    • Uncertain numbers may be related to themselves over time… e.g.
      • Interest rates high over a period of time are more likely to go down.

• Decisions and information

  • If information can’t impact a decision, it’s worthless.
  • People aren’t rational.
  • Decision trees
    • What chance of the uncertain event would make you change your decision? It is hard to arrive at a firm probability but easier to agree on whether it’s greater or less than a given probability.
  • The value of information

• The sins of averaging

  • The family with 1.5 children. Often the average scenario is nonexistent.
  • Why everything is behind schedule. Setting each task at its average results in project completion in 6 months, but the chance that all ten come in at their average or sooner is the same as flipping ten heads in a row, so the chance of finishing by 6 months is less than one in a thousand.
  • **The egg basket. **diversification; ten eggs in the same basket as opposed to one each in separate baskets. If there’s a 10 percent chance of dropping a basket the average of is coming away with 9 eggs, however in the first you’ve got 10 percent chance of loosing all your eggs whereas in the second you have a one in 10,000,000,000 of losing all the eggs.
  • The risk of ranking. Portfolio theory is based on the interdependence of investments, so simply a best to worst ranking won’t do. Fire insurance seems a bad investment as on average it looses money. But it doesn’t look that way if you’ve got a house to go with it.
  • Ignoring restrictions. Beware of a downside without an associated upside. Average profit will be less than associated average demand.
  • Ignoring optionality. Find upside without associated downside. Gas production can stop if the price drops too low, limiting downside.
  • The double whammy. The cost associated with average demand is zero, but average cost is positive because demand is never average.
  • The flaw of extremes. In bottom up budgeting, reporting the 90th percentile of cash needs leads to ever thicker layers of unnecessary cash as the figures are rolled up to higher levels.
    • The smaller the sample size, the greater the variability of the average of that sample. Highest average ear lobe size and lowest average as found in small towns.
    • Sandbagging is when budgets are inflated to numbers that individuals are sure they won’t exceed (90th percentile). When independent uncertainties are added together, diversification causes the distribution to get relatively narrower. In the case of budgeting, this is a waste, the CEO doesn’t need that level of certainty.
  • Simpson’s paradox. Can you imagine a nutritional supplement that on average causes people to loose weight?
    • Occurs when variables depend on hidden dimensions in the data.
  • The Scholtes revenue fallacy. Suppose you sell different quantities of various types of product, each with it’s own profit per unit. You might make a nice profit on your average product but a loss overall. You need to take account of statistical relationships.
  • Taking credit for chance occurrences. Some success may be due to dumb luck.

• Applications

  • Finance
    • Portfolio modelling
    • Options

• Hot button issues

  • The Statistical Research Group of WW2
  • Probability and the War On Terror
  • Climate Change
  • Health Care
  • Sex and the central limit theorem

• Probability Management