Tom Webster, writing and speaking

Size Doesn't Always Matter

Added on by Tom Webster.

Chocoladetruffels LindtA couple of days ago someone forwarded a survey to me that made a mildly preposterous claim, but it was backed by a sample size well north of 50,000. "I didn't believe it," my friend noted, "but that's a big sample. Must be true...right?" Well, not exactly. Sample size plays a role, but what is more important is how representative that sample is. If you sample a population truly randomly (such that every member of that population has an equal, non-zero chance of being sampled), then you don't need thousands of people to take a pretty credible stab at a number. If you throw up a web poll on Twitter and get 10,000 responses, you've got...10,000 responses. It's not nothing, but size doesn't save it. Similarly, a sample of 80 sounds small, unless it's 80 Fortune 500 CEOs. In that case, 80 people is a pretty damn good sample.

Here's best example I can think of from the airport: Let's take a question like "Which do you love more, chocolate, or the leader of your country?" Ask 400 3rd graders and you'll get "chocolate," guaranteed. One million North Koreans will have a different answer. In this case, as in many others, finding the truth is less about how many you sampled, and a lot more to do with who you asked.

Surely even Beloved Leader knows the right answer to this one.