Statistics saturate our lives. Percentages plaster our billboards, gross numbers leap out at us from headlines, and weather probabilities fill our morning and evening news. But statistics can be tricky—their credibility depends on a lot of factors, like context and the validity of the underlying data. How can you tell whether the numbers you read truly shed new light on the world, rather than simply paint it in the shades that companies or special interests want you to view it in? Paying attention to a few key factors can help us become sophisticated connoisseurs of the data around us.
Thirty percent of statistics are made up on the spot. (Like this one.) More commonly, statistics are true but specific to a certain context—they become unreliable when writers unwittingly transplant them to a different situation. Identifying faulty data and appropriately interpreting its applications become crucial. Below are a few key things to help you better interpret everyday statistics you encounter, adapted from Joel Best’s Stat-Spotting: A Field Guide to Identifying Dubious Data.
Check the sample population. This is critical for two reasons. On one hand, you want to be clear on who the statistic really represents. On the other hand, you need to understand the magnitude of a percentage or the proportional weight of a gross number.
- A New York Times article I read recently stated: “For the second time in a week, Ben Carson took the lead in a national poll, drawing 29 percent support in an NBC/Wall Street Journal survey and eclipsing Donald J. Trump, who had 23 percent.” We know the survey is “national,” but who did they really poll? Digging deeper, we find the survey drew responses from 1,000 individuals via phone, and the question was asked only of registered voters who said they would vote in the Republican party. The sample size alone is not concerning—properly drawn samples can reliably represent large populations—but it is important to note that the sample represents only currently registered Republican voters who have reliable access to a phone.
Double check the calculation. People do make mistakes, and miscalculations or misplaced decimal points occur, even in formally published data. If a statistic seems outlandish, double check the “benchmark” (population size) it purports to represent, and run some basic calculations to see if the numbers really add up.
- Best’s book describes the headline of a book advertisement that read: “Today, a young person, age 14-26, kills herself or himself every 13 minutes in the United States.” This seems unrealistically high, and it turns out that it is. There are 525,600 minutes in a year (365 days x 24 hours x 60 minutes), and, at the time of the advertisement’s publication, 4,010 suicides per year (National Vital Statistics Reports). This works out to a suicide every 131 minutes, not every 13.
Not all large numbers are equal. Our brains tend to interpret all numbers above 100,000 as “a lot.” One million, though, is very substantively different from a billion or trillion; all three are made up of equally meaningful, individual units. Sometimes the reality statistics try to convey can be lost to the sheer magnitude of the numbers they employ.
- In a study conducted through the University of Pennsylvania, researchers found that people are more likely to donate to individuals than they are to donate to large groups of people. They found that people lose their sense of emotional connection to a multitude of people, even if their money could have impact on more individual lives by donating to that multitude.
These tips can help you better navigate the increasingly numeric world that we live in. Always remind yourself that a number doesn’t ensure a fact’s reliability, but some simple probing and mental cautioning can help you distill the “noise” from the meaningful data that surrounds us.