Number Sense & Nonsense

you will know from the news media, business executives and the techno-sphere,
we are in the age of big data and analytics.
(Disclosure: I too am part of this trend with my forthcoming course on
leading change in the Applied Analytics Master’s program at Columbia University.)

those of us who have been practitioners of analytics, this attention is long
overdue.  But there is a certain naiveté
in the breathless stories we have all read and in many of the uses – really misuses
– of analytics that we see now.

Partly to provide a more mature understanding of analytics, Kaiser Fung,
the director of the Columbia program, has written an insightful book
titled “NumberSense”. 


Filled with compelling examples, the book is a
general call for more sophistication in this age of big data.  I like to
think of it as a warning that a superficial look at the numbers you
first see will not necessarily provide the most accurate picture, any
more than the first thing you see about an unpeeled onion tells you as
much as you can see once it is cut.


Continuing this theme in his recent book, “The End Of Average”, Todd
Rose has popularized the story of the Air Force’s misuse of averages and
rankings after World War II.  He describes how the Air Force was faced
with an inexplicable series of accidents despite nothing being wrong
with the equipment or seemingly with the training of the pilots.  The
Air Force had even gone to the effort of designing the cockpits to fit
the exact dimensions of the average pilot!


As Rose reports in a recent article:

the early 1950s, the U.S. air force measured more than 4,000 pilots on
140 dimensions of size, in order to tailor cockpit design to the
‘average’ pilot … [But] Out of 4,063 pilots, not a single airman fit
within the average range on all 10 dimensions.  One pilot might have a
longer-than-average arm length, but a shorter-than-average leg length.
Another pilot might have a big chest but small hips.  Even more
astonishing, Daniels discovered that if you picked out just three of the
ten dimensions of size — say, neck circumference, thigh circumference
and wrist circumference — less than 3.5 per cent of pilots would be
average sized on all three dimensions.  Daniels’s findings were clear
and incontrovertible.  There was no such thing as an average pilot.
If you’ve designed a cockpit to fit the average pilot, you’ve actually
designed it to fit no one.”

Rose criticizes the very popular
one-dimension rankings and calls for an understanding of the full
complexity, the multi-dimensional nature of human behavior and
performance.  As a Harvard Professor of Education, he puts special
emphasis on the misleading rankings that every student faces.

He shows three ways that these averages can mislead by not recognizing that:

  1. The
    one number used to rank someone actually represents multiple dimensions
    of skills, personality and the like. Two people can have the same
    score, but actually have a very different set of attributes.
  2. Behavior and skill change depending upon context.
  3. The
    path to even the same endpoint can be different for two people. While
    they may look the same when they get there, watching their progress
    shows a different picture.  He provides, as an example, the various
    patterns of infants learning to walk.  Eventually, they all do learn,
    but many babies do not follow any standard pattern of doing so.

is not too difficult to take this argument back to Michael Lewis’s
portrayal in Moneyball of the way that the Oakland A’s put together a
successful roster by not selecting those who looked like star baseball
athletes – a uni-dimensional, if very subjective, ranking.

hope that as big data analytics mature, there are more instances of
Moneyball sophistication and less of the academic rankings that Rose

© 2016 Norman Jacknis, All Rights Reserved


Leave a Reply

Your email address will not be published. Required fields are marked *