This morning, I read the following in an email newsletter I receive regularly from a research company:
68% of consumers surveyed…say they order carry-out pizza once per month compared with 24% in 2010.
To the casual reader, this might be interpreted as: “More consumers are ordering carry-out pizza now than two years ago.” This may very well be true.
BUT, the reverse may also be true. What that data point fails to convey is what is happening to the other consumers surveyed. There are 32% (100%-68%) of consumers that order carry-out pizza either never, or perhaps as many as 2, 3, or more times a month.
So, it may actually be the case that 32% of consumers are ordering carry-out pizza 2 or more times per month now, compared with 76% (100% – 24%) in 2010. In other words, more people are ordering fewer carry-out pizzas now. So, the total number of carry-out pizzas sold is actually less.
So, which is it? Is there more or less carry-out pizza being ordered?
We can’t tell from the data.
And, to be perfectly honest, since I am not in the pizza business, I don’t really care, and you probably don’t either. But what you should care about is whether you are getting the full picture from the research you conduct or purchase and whether you are making the right or wrong decisions based on that research.
So, when you are looking at research reports, or are simply reading articles or newsletters, remember these tips to make sure you get the full story:
1) What is this data NOT telling me? Think about the converse of the data being presented to help identify whether there is important data missing preventing a full picture.
2) What is the sample size and methodology? The exact same survey can produce very different results when different samples are utilized. Users of a brand or product tend to have very different responses than non-users. Young respondents will answer differently than old. It is critical to define the target market and have crystal clear objectives of what you want to learn to get the sample and methodology right.
3) What is the objective of the writer? Is there a chance they are biased and therefore, intentionally slanting the data being presented? Showing a short date range instead of a long one can mean the difference between a few isolated data points moving in a particular direction or a real long-term trend. The scale on a graph can make a big difference in the interpretation of data in a chart. It may not be terribly ethical, but it is not hard to manipulate data to make a point.
I don’t mean to insinuate that all people reporting, interpreting, or commenting on data are intentionally trying to mislead you. That is the exception, rather than the rule. But even with the best of intentions, data can inadvertently be misrepresented, even by experts. Just review everything with a critical eye and ask questions if something doesn’t sound quite right!
Sift Cipher & Bloom