Tom Webster, writing and speaking

You're Doing It Right: A Great Methodology Statement

Added on by Tom Webster.

I've written plenty on proper methods and practices for vetting survey data (especially in the social media space, where there is so little great data to be had.) Many studies fall down in the methodology reporting area - no methodology is perfect, but the worst thing you can do is to simply gloss over your methods and not provide a proper methodology statement. Those are surveys I ignore. So I was really pleased to see the methodology statement for this ExactTarget / CoTweet research series (hat tip to Jay Baer for alerting me to the study.) Setting aside all the other specific aspects of the sampling, here are the three things that warmed my heart the most:

1. They began a quantitative survey with an initial qualitative project to generate questions, ideas and language. I am a huge proponent of this - it often gets shot down in the budgeting process, but no one is more qualified to provide the language for a survey than the population you are surveying. In my day job, I actually spend more time doing qualitative research than quant work. Irreplaceable.

2. They were specific about stratification, size of strata (no fewer than 200 persons) and the time of fielding. By the way, the most common critique of the uninitiated is to simply remark "sample size." Give me 200 properly sampled, representative and well-distributed respondents over 10,000 self-selected people-who-saw-your-survey-link-on-Twitter any day of the week. When sampling methods are poor or nonexistent, more is not better. More is more worser.

And don't even get me started on that other oft-cried caveat, margin of error.

3. They weighted the data to accepted sources to mirror the U.S. population. Crucial step here, and one that, oh, 100% of self-selected web polls never attempt.

Kudos to ExactTarget and CoTweet not just for a great series, but a model methodology page.