BrandSavant

Gaining Insight From Social Media Data

Sometimes I Hate Market Research

by Tom Webster on June 1, 2010

Maybe that’s a little strong. Here’s what I do take issue with, however – misrepresenting an impressive-sounding sample size as a valid study. Today I got a pitch about a “social media study” that purported to have a sample size of over 2,000, which if done correctly would have a margin of error of just +/- 2.0%. I won’t dignify this particular study with a link, but a closer inspection of the methods revealed this:

Sample size: 2322 persons 6-54, Error +/- 2.0%
Persons ages 6-54, 55% male/45% female
Address-based sampling
The study was conducted between January 15 and April 30, 2010.
Data was accumulated through in-person interviews, telephone interviews, social media focus groups and web surveys.

See, if you saw the results from this study and didn’t bother to scrutinize the methods, you might be fooled into thinking you had a good piece of research to write about. Sometimes, sample size can fool you, as it does in this case. Consider:

1. The data was accumulated over almost four months. This introduces a longitudinal bias into the data. Think opinions have changed about BP over four months?

2. The data was collected multi-modally, using a series of qualitative projects all cobbled together. You can only really claim to have a given sample size for a survey if it is the same survey. Much of this data appears to have been quantitative questions asked in multiple small groups, and then simply crunched together. If the sample really was 2300 persons surveyed in one mode (phone, web survey, etc) then you could claim that the margin of error is +/- 2.0%. If you just add together a bunch of tiny, unrepresentative qualitative samples, all you are really doing is compounding error. A typical focus group, for instance, might have 10 respondents. The margin of error on a 10-person survey is, I dunno, about +/- 99% :) You can multiply those tiny samples together all you want – they don’t get any better.

3. 55% Male/45% Female is another way of saying “convenience sample,” or, “this is who showed up for the focus groups.” The US population is actually slightly more female.

4. Where were these interviews conducted? It would be pretty hard to do a nationally representative sample using in-person interviews and focus groups. Certainly you’d make Southwest Airlines pretty happy. There is surely an unstated, unknown geographic bias. Probably a pretty sizable one.

5. What’s a “social media focus group?” If such an animal exists, it needs some esplainin’.

Basically, you can’t claim this sort of “survey” is representative of anything, even if the sample were 10,000. What’s worse, you can’t characterize the sample at all, so you can’t even refer to the data as “amongst respondents to this survey.” Which survey?

So, this is a rare airing of sour grapes here on BrandSavant, but my purpose here is to get you to think about numbers when you see them. Surveys like this are done all the time – it’s up to you to ask the tough questions before you report on the results or – heaven forbid – act on them.

Tomorrow, a more positive post. :)

Be Sociable, Share!
Buffer
  • http://www.ddmcd.com Dennis McDonald

    One thing to watch for in a report like this is the slide that really tells what went on. You may see such a slide in the initial report from the research groups to the project’s research manager.

    It gets edited because explaining things always results in too much detail.

    Then it gets some more editing when the client’s project manager gets hold of it. That slide keeps getting shorter. Now it’s an appendix. “Management just wants the findings.”

    Finally the client’s management gets to see a dry run of the presentation along with suggestions for a press release. What was once a moderately informative dry reporting of how the study was actually conducted has now become a polished, smooth, delicately phrase piece of, um, uncommunicative gibberish that is designed to hide inconvenient facts that boil down to, “Hey, bad information is better than no information at all!”

    Especially when it confirms someone’s preconceived notions.

  • Tom Webster

    You are exactly right, Dennis. One of the most conceptually difficult things for some clients to understand is that bad research is potentially worse than no research.

  • http://www.ddmcd.com Dennis McDonald

    Yes. But it’s also true that some clients are more willing to pay for bad research than no research.

  • http://www.thehumanelement.co.uk Alison

    That is just horrible. Was the age really 6-54 or is that a typo (I hope it’s a typo!)

    I bet you a social media focus group is otherwise known as a bulletin board.

  • Tom Webster

    Sadly, it was not a typo. There were “results” reported for 6-12 year-olds.

    I never thought to call a bulletin board a “social media focus group.” BRILLIANT!

  • Damian Vanderwolf

    It is pretty clear there were a few flaws in the research design. I do though have a pragmatic question:
    Do you believe that something can be salvaged from this? Apart from lessons about research design, do you think this data could be of any value to the client at the end of the day?

  • Tom Webster

    I suspect this particular “dataset” was cobbled together from multiple projects for multiple clients. I am sure that those clients got some insight from those individual qualitative projects.

    But mashing them together into one multi-modal, long time-frame, mixed-sample mess? Worse than doing no research at all.

  • Dee

    As a Project Manager in a Market Research firm these numbers are appalling. There is no way a consumer study for a Media based project would interview anyone less than 18 years old. The majority of media projects use what’s called the ‘birthday method’ and look to speak with the person in the household with the most recent birthday’.\n\nTo combine qualitative and quantitative interviews together like that just seems unreal to me. The tabulations for this study must have been an absolute mess. \n\nMost projects done on a consumer level break the male/female ratio to 60 (female) 40 (male). Also, we usually cap the 65+ age group at 35% of the sample size.\n\nThis Market Research company should be assumed of themselves providing this client with such poor data.

  • Anonymous

     To continue with the “Survey/Market
    Research” theme: (Nice blog spot here by the way)

    Honestly, every post that you see on the web
    describes the next BIG methodology when it comes to B2B Market Research/B2B/B2C
    Surveys. At my company Partner Source, which is located in Minneapolis, Minnesota,
    we approach the B2B Market Research/B2B/B2C Surveys subject with science, as it
    is our business. We have taken over 20 years of B2B Market Research/B2B/B2C
    Surveys historical data and the proof is in the pudding, you can’t argue the
    results.

    Here is what we offer, what do you think about
    this position?

    Telephone surveys from Partner Source can help
    your organization:
    • Obtain valuable feedback from hard-to-reach
    groups
    • Make intelligent business decisions
    • Acquire complete and reliable data

    Who has Partner Source provided this service for?

    We have successfully created and delivered survey
    campaigns for government agencies and businesses from around the world. If you
    are looking to better understand your customer or simply need to gather
    information from the population, we are your go-to provider. We have the
    ability to serve any customer, no matter the size, as we have delivered
    campaigns ranging from very complex to basic and simplistic.

    Telephone Research Surveys – An Important
    Methodology

    Savvy researchers understand telephone surveys
    still play an important role in market research, despite the growing popularity
    of online surveys.
    Telephone research surveys can be the ideal
    methodology for organizations that need to gather detailed data from
    hard-to-reach B2B, B2C, or customer groups. Through telephone surveys, your
    organization can:

    Sample at a local market level
    Reach specialized B2B contacts
    Connect with hard-to-reach customers
    Control the sequence in which survey questions are
    asked.
    Use probing techniques to obtain in-depth answers
    to open-ended questions.
    Ensure that all questions are answered.
    Gather data on more sensitive topics by
    alleviating some of the discomfort and reluctance respondents might present
    during a face-to-face interview.
    Clarify and substantiate unclear responses.
    Ask follow-up questions.

    Design, Administration and Analysis of Telephone
    Surveys

    At Partner Source, our goal is to equip your
    organization with bias-free telephone research that generates valid data and
    high response rates. All of our telephone surveys include the following steps:

    Design – Partner Source’s experienced researchers
    will work with you to design customized telephone surveys based on your
    specific objectives. We can work within our proven telephone survey templates,
    or can create entirely new telephone surveys “from scratch.”

    Administration – Using our proprietary methods,
    Partner Source takes your telephone surveys and administers them to your
    desired recipients. The interviewers adhere to best practices, ensuring data
    collection that’s free from bias while heeding to our clients’ unique
    requirements for security level, deadlines, incentives, etc. Throughout the
    administration phase, your data is made available online through Partner
    Source’s Real-Time Report Menu.

    Analysis – The raw data garnered from your
    telephone surveys is thoroughly analyzed and converted into actionable
    conclusions. We offer a variety of analysis packages for every budget.

    Our Team: The entire call team for these projects
    is located right here in Minneapolis,
    Minnesota, USA.
    We do have many additional resources if other languages are required besides
    English.

    Glenn Wright
    Partner Source (B2B Surveys & B2C Surveys
    (Phone Surveys/Market Research))
    Minneapolis, Minnesota
    http://www.thepartnersource.co…

Previous post:

Next post: