Tom Webster, writing and speaking

You Got It All Wrong: The Limits of Social Media Monitoring

Added on by Tom Webster.

You got this one wrong, I'm afraid. You tried to predict the winners of last night's Grammy Awards, but you were more than a little off. According to the folks at Meltwater Buzz (see the chart, below), your social media conversations indicated that you thought Eminem would get Album of the Year, the Bieber would be Best New Artist, and the Record of the Year would be "F*** You," by Cee Lo Green (this is a family blog, now.) In fact, not only did you get this one wrong, you got it really wrong. Bieber over Esperanza Spalding? Eminem over Arcade Fire? How could you have screwed up Best New Artist so badly? :)

Well, it isn't really your fault. I'm also not going to beat up on the tool - it was built to track mentions, and I have no reason to believe that it didn't do an admirable job (so, Meltwater folks, stand down - this isn't about you.) The issue here is that you were asked the wrong question. Tracking mentions for predictive purposes is of dubious value - in the short term, the volume of mentions is a random walk.

The Grammy voters are a notoriously inscrutable bunch (witness the Best Heavy Metal Album award given to Jethro Tull a few years ago. Really, dude? Really??) Had you been asked, "What act do you think will win Best New Artist at the Grammys?" you might have taken that inscrutability into account and hazarded a different guess from The Bieber. You weren't asked, however. Instead, you were asked to "name a band." Wrong question, wrong result.

Therein lies the central difference between Social Media Monitoring, and Social Media Research. In monitoring, the "mention" is a core metric - a trigger for action. In research, however, the quantity of mentions might not even be relevant.

Now, you might accuse me of being a little unfair here - in no way should social media mentions correlate with the votes cast by a small, closed community. No one would honestly claim that mentions are predictive of the Grammys, or the Oscars. We are, however, just a few short months away from the 2012 election cycle, and, as we have seen in the past, the level of social mentions for some candidates may, on the surface, approximate their actual support at the polls. It won't be true of all candidates, however, and the extent to which it is or isn't true will be unpredictable and random. Yet, the articles about these analyses will inevitably focus on the close calls, and ignore the "bad beats," as they say in Vegas. Again, monitoring is monitoring, and research is research. We mustn't allow ourselves to be fooled.

Trying to ascertain the winners of the Grammy awards by counting mentions is using a sledgehammer, when a scalpel was called for. The tools are getting better - and the scalpels are just around the corner - but we'll never get the right answers from social media unless we ask the right questions.

Grammy Infographic v2