Maybe that’s a little strong. Here’s what I do take issue with, however – misrepresenting an impressive-sounding sample size as a valid study. Today I got a pitch about a “social media study” that purported to have a sample size of over 2,000, which if done correctly would have a margin of error of just +/- 2.0%. I won’t dignify this particular study with a link, but a closer inspection of the methods revealed this:
Sample size: 2322 persons 6-54, Error +/- 2.0%
Persons ages 6-54, 55% male/45% female
Address-based sampling
The study was conducted between January 15 and April 30, 2010.
Data was accumulated through in-person interviews, telephone interviews, social media focus groups and web surveys.
See, if you saw the results from this study and didn’t bother to scrutinize the methods, you might be fooled into thinking you had a good piece of research to write about. Sometimes, sample size can fool you, as it does in this case. Consider:
1. The data was accumulated over almost four months. This introduces a longitudinal bias into the data. Think opinions have changed about BP over four months?
2. The data was collected multi-modally, using a series of qualitative projects all cobbled together. You can only really claim to have a given sample size for a survey if it is the same survey. Much of this data appears to have been quantitative questions asked in multiple small groups, and then simply crunched together. If the sample really was 2300 persons surveyed in one mode (phone, web survey, etc) then you could claim that the margin of error is +/- 2.0%. If you just add together a bunch of tiny, unrepresentative qualitative samples, all you are really doing is compounding error. A typical focus group, for instance, might have 10 respondents. The margin of error on a 10-person survey is, I dunno, about +/- 99% 
You can multiply those tiny samples together all you want – they don’t get any better.
3. 55% Male/45% Female is another way of saying “convenience sample,” or, “this is who showed up for the focus groups.” The US population is actually slightly more female.
4. Where were these interviews conducted? It would be pretty hard to do a nationally representative sample using in-person interviews and focus groups. Certainly you’d make Southwest Airlines pretty happy. There is surely an unstated, unknown geographic bias. Probably a pretty sizable one.
5. What’s a “social media focus group?” If such an animal exists, it needs some esplainin’.
Basically, you can’t claim this sort of “survey” is representative of anything, even if the sample were 10,000. What’s worse, you can’t characterize the sample at all, so you can’t even refer to the data as “amongst respondents to this survey.” Which survey?
So, this is a rare airing of sour grapes here on BrandSavant, but my purpose here is to get you to think about numbers when you see them. Surveys like this are done all the time – it’s up to you to ask the tough questions before you report on the results or – heaven forbid – act on them.
Tomorrow, a more positive post.
{ 8 comments… read them below or add one }
One thing to watch for in a report like this is the slide that really tells what went on. You may see such a slide in the initial report from the research groups to the project’s research manager.
It gets edited because explaining things always results in too much detail.
Then it gets some more editing when the client’s project manager gets hold of it. That slide keeps getting shorter. Now it’s an appendix. “Management just wants the findings.”
Finally the client’s management gets to see a dry run of the presentation along with suggestions for a press release. What was once a moderately informative dry reporting of how the study was actually conducted has now become a polished, smooth, delicately phrase piece of, um, uncommunicative gibberish that is designed to hide inconvenient facts that boil down to, “Hey, bad information is better than no information at all!”
Especially when it confirms someone’s preconceived notions.
You are exactly right, Dennis. One of the most conceptually difficult things for some clients to understand is that bad research is potentially worse than no research.
Yes. But it’s also true that some clients are more willing to pay for bad research than no research.
That is just horrible. Was the age really 6-54 or is that a typo (I hope it’s a typo!)
I bet you a social media focus group is otherwise known as a bulletin board.
Sadly, it was not a typo. There were “results” reported for 6-12 year-olds.
I never thought to call a bulletin board a “social media focus group.” BRILLIANT!
It is pretty clear there were a few flaws in the research design. I do though have a pragmatic question:
Do you believe that something can be salvaged from this? Apart from lessons about research design, do you think this data could be of any value to the client at the end of the day?
I suspect this particular “dataset” was cobbled together from multiple projects for multiple clients. I am sure that those clients got some insight from those individual qualitative projects.
But mashing them together into one multi-modal, long time-frame, mixed-sample mess? Worse than doing no research at all.
As a Project Manager in a Market Research firm these numbers are appalling. There is no way a consumer study for a Media based project would interview anyone less than 18 years old. The majority of media projects use what’s called the ‘birthday method’ and look to speak with the person in the household with the most recent birthday’.\n\nTo combine qualitative and quantitative interviews together like that just seems unreal to me. The tabulations for this study must have been an absolute mess. \n\nMost projects done on a consumer level break the male/female ratio to 60 (female) 40 (male). Also, we usually cap the 65+ age group at 35% of the sample size.\n\nThis Market Research company should be assumed of themselves providing this client with such poor data.