Tom Webster, writing and speaking

Google Consumer Surveys: Are You Feeling Lucky?

Added on by Tom Webster.

Yesterday, I observed the following factoid in the wild (AKA, The Twitter): Wikipedia Data Tweet

Now, I will admit to some previous discomfort with the sampling methodology used by Google Consumer Surveys. Their model places questions on partner sites, requiring readers to answer a single question before they can access premium content. Demographic data is inferred, basically, through an algorithm based upon browsing history. The Achilles' heel of this methodology is not necessarily response rate (which seems to be decent) but where these questions are placed. There is no way around the fact that if there is some bias to the types of content or sites Google chooses for question placement, the data will be biased in significant and inscrutable ways.

Now, I'm fully aware that they are iterating and continually improving, as well as building out their network of content partners for question placement. However, this Wikipedia data point, as Tom Ewing pointed out in response, is a total "unicorn stat." I think I was a little less charitable. Here's the exact data point:

Survey swaqqdis3i32i question 1

There is of course no way on the planet that one in eight Americans has edited a Wikipedia article. I'll get to just how far off the mark this is in a moment, but suffice it to say it's off by a crapton. In my day job, we're not just market researchers, we're pollsters--so we take things like "…of Americans" very, very seriously.

It turns out that the tweet was out of context, as fellow market researcher Leonard Murphy of GreenBook pointed out--Google themselves make no claim that the research represents "Americans" (which is axiomatic, given that nearly 15% of Americans aren't even online) and that Google is up front about the biases inherent in their data.

Google does indeed list some of the biases in their data, right below the results:

Sample bias

It's a detailed chart, and allows the careful researcher to deconstruct how Google has weighted the data to make it more "representative" of the online population. It's so compelling, in fact, that it might lead the casual observer to conclude that these data are, in fact, representative and reliable.

But let me repeat the finding here: One in eight respondents have edited a Wikipedia page, and Google does label the sample as the "National adult Internet population." Fluctuations in age, sex and region don't even begin to cover the bias inherent in this corker.

Wikipedia itself offers some clues to the potential scale of this number. There are, for example, about 300,000 people--worldwide--who have made at least 10 edits. Also, 20% of Wikipedia editors are from the US, and 91% of editors are male (lets get busy out there, ladies!). Wikipedia also notes in a 2011 study of the editing community that the number of editors has flattened, and the number of active (>10 edits/month) has declined significantly.

Of course, you don't have to be a registered "editor" to edit Wikipedia pages--you can do so anonymously, which throws a spanner into the works. However, it turns out that the number of anonymous editors is far lower than the number of registered editors, and currently sits at about 20% of the editing base (and, like the number of active editors, has declined significantly.)

All of this means that the number of unique Americans, cumulatively over time, who have edited a Wikipedia page is likely in the low six figures. The Google Consumer Surveys number isn't just a little off, it might be off by a factor of 100.

This enormous bias can't be explained by age, sex or region--in fact, characterizing this bias is probably a doctoral thesis waiting to be written. The "convenience sampling" methodology of placing questions on partner sites results in a slanting of the data that cannot readily be quantified. There are two possibilities here. Either there is a significant, repeatable bias to Google Consumer Surveys data (which, like the stopped clock, would at least be right some of the time) OR there is a significant, variable bias to the data--i.e., this Wikipedia question was off, but asking whether or not you drink tea might be close to right. This is actually the more disturbing of the two possibilities, because it essentially comes down to luck. Are you feeling lucky?

To their endless credit, the GCS team is iterating, and listening. In fact, they responded to me on Twitter yesterday with this:

Google response

Kudos for their openness and willingness to challenge their data--and again, I don't doubt that GCS will continue to improve. I was left wondering, however, what the result of running this additional data would be, and again, there were two possibilities: either they would come back with the same answer (bad) or they would come back with a different answer (really, really bad)

What they came back with was somewhere in between the two. They ran an additional screening question to isolate actual Wikipedia users, and have revised their estimate to "less than 5% of the US Internet" population, which is aggressively true. :)

So, what do we do with Google Consumer Surveys? I love the model--I really do--and want to be able to use it to answer quick, one-off questions cost-effectively and rapidly. It has the potential to be a great service, and I am certainly not suggesting they won't get there, someday. But the smartest thing I can say about it today is that I can't make head nor tail of the data it is spitting out on questions like this.

And if I don't understand it, I can't use it.

I will admit to not having a firm position on this one yet, and your comments can certainly sway me one way or the other. What's your take? Is this a fair evaluation? Or have I missed something germane? Fire away.