BrandSavant

Gaining Insight From Social Media Data

“Unaided Recall” in Social Media Research

by Tom Webster on August 30, 2010

Some of the most commented-upon posts here at Brandsavant have been about one of two topics: sentiment analysis, and social media monitoring. I suspect there are three reasons for this:

  1. There are lots of sentiment analysis folks analyzing sentiment for sentiments about sentiment analysis
  2. There are lots of social media monitoring folks monitoring social media for media about social media monitoring
  3. ….uhh…I got nothin’.

Anyway, if this popped up in your social media monitoring dashboard, welcome :) .

One thing I’ve been thinking about recently is the process of parsing through comment threads, and how modern listening and analysis services could actually follow a sequential conversation that might take place across multiple sites and services. When I first started poking around sentiment analysis, I got loads of comments, for which I am very grateful, and some wonderful conversations started up in and around the space of that initial post.

If you were using some kind of automated monitoring and sentiment analysis software to go through that post and the comments that followed, you’d certainly have no shortage of grist for your mill. Yet, if that conversation were parsed by a machine, one thing that might be lost in your analysis is this subtlety – I poked that particular hornet’s nest. All of the “sentiment” about sentiment analysis in the comments below that post were prompted – I provoked others to leave opinions that might have otherwise never surfaced online.

In my day job, we do a lot of marketing/advertising effectiveness work for brands. This entails a lot of pre- and post-campaign measurement of basic measures like awareness or intent to purchase, various metrics for engagement, and even things like “opportunity to see” for our digital out-of-home advertising clients. In almost all cases, we construct a survey that reads from general to specific, leading respondents through these various measures in a planned and logical order. Two bread-and-butter metrics for any kind of brand research are unaided recall and aided recall. An “unaided recall” question occurs near the very beginning of a survey: “Tell me all the brands of luxury automobile you can name.” This sort of question is asked at the start of the process to see which brands have true top-of-mind awareness, and which ones don’t. Later in a survey, we might get more specific: “Have you ever heard of ‘Maybach.’” At the end of such a survey, we might even ask specific questions about the Maybach (“how much do you think it costs,” etc.) or ask respondents to rank the Maybach amongst other brands on various attributes. Again, this is real meat-and-potatoes stuff, but it’s absolutely essential to track, especially if you want to measure the “lift” of a given campaign.

To get back on topic, if I write a post about sentiment analysis or social media monitoring, and I happen to name a particular service, then the folks monitoring for those keywords will dutifully tally this up as a “mention” (which is, as I’ve noted previously, a dubious metric anyway.) However, it’s actually a little bit more than that – it’s an unaided mention. In other words, from a cold start, I named that particular site or service unprompted. That mention has a slightly different value than a mention later in a comment thread by people responding about that particular brand and leaving their opinions. Those mentions, in the context of that post, constitute aided recall – I prompted each and every comment about that brand, and many of those mentions/opinions might never have surfaced had I not, again, prodded the nest.

All of which leads me to a very interesting social media research question: do you track unaided vs. aided mentions? Top-of-mind opinions vs. prompted opinions? Do you see the value in those measures?

P.S. I do not drive a Maybach. If the folks at Maybach would like to send me one to “review,” I promise to disclose the arrangement, deliver a fair review, and then abscond with the car, never to be seen again. You have been warned.

Buffer
  • http://rickcaffeinated.com Rick

    It might be too early in the morning for this, but thanks for the thinks :) . I think you’re spot on – what you mention has value to those listening, but the nuances for what that mention really means be hard to decipher, and subsequent mentions would degrade from there somehow, I’d think. Hard to pinpoint that mathematically.

  • Tom Webster

    Well, it’s afternoon somewhere, Rick. Though not where you and I are writing from, I suspect, so it was probably a bit too early in the morning for me, too :)

  • http://socialfresh.com Jason Keath

    So an evidentiary chain of custody? Feel free to retitle this post “Brand Monitoring CSI”.

    But it makes sense. Especially if you want to monitor the conversation threads across platforms as you suggested. I see value there. Finding root influencers, tracing how conversation evolves into action, etc.

    Two caveats. 1) I cannot imagine 99% of companies (people) that are monitoring conversations will choose to dive this deep. For better or worse. 2) Isn’t every brand mention prompted on some level. I suppose you are talking about a chain of custody of sorts here that is purely online. So many brand mentions will originate from some action offline. Another wrinkle to the always murky marketing equation as usual.

    Overall I think this would be very cool to explore for a brand that is ready to act on it.

  • Tom Webster

    Jason, I think you should just write all my headlines from now on, whether they are on socialfresh.com or here :)

    About your caveats: First, you are correct about #1. That’s why it’s worth it – because it’s difficult. But if “social media research” is going to be actually useful as a source of consumer insights and decision support, it can’t be ignored.

    As far as # 2 is concerned, not necessarily. This very post is a great example. Nothing “prompted” me to use the Maybach brand – it was the first super-high end auto brand I thought of. In traditional market research, we track that sort of thing. If you then write a post on your blog that links to mine, and in your rebuttal/supporting post you also mention Maybach, then it’s arguably because I planted the seed.

    I’m still not getting that Maybach, it seems, but I’ll keep trying.

  • http://blog.sysomos.com 40deuce

    This is a great point you make here Tom. It’s probably one that many people have never even thought of.
    It is quite a hard thing to do, especially if you’re using a tool to monitor brand mentions. I like to personally try and track mentions to where the conversation started, but that’s not always possible, and I do it manually once I pick up a mention through our software. I don’t do it to qualify an aided mention vs a unadied mention, but I do see why that would actually be a useful thing to do.
    Now that you bring it up, I wonder how many people are measuring for this? I do see it being something that some may find very useful and interesting. Although, I’m sure that now most brands are just looking for all mentions in general, but this could be a very big metric in the future.
    Thanks for making me thing about this.

    Cheers,

    Sheldon, community manager for Sysomos

  • http://www.eastridgeprint.com Renee Malove

    First and foremost, I would like to extend my sincerest regrets for the continued absence of your Maybach. I will continue to hope that they completely disregard the last portion of your post and submit to an obviously superior method of MRX by handing over the keys. This is, of course, with the understanding that my high recommendation of your character as someone who would never swipe a test car and head south of the border entitles me to at least one test drive before you take off!

    That being said, thanks for this. Trying to explain the difference between aided recall and unaided recall can be incredibly difficult, especially with the apparent ambiguity of brand ownership these days-for example, someone talking about “ziploc bags” might not actually be talking about “Ziploc bags” but rather some form of generic alternative or similar product. The brand has just become so associated. However, this simple error can lump the association firmly under the heading of “aided recall”, which means the results you get back might not be quite what you’re looking for.

    I’m not 100% certain that was the best example, but you see where I’m going with this!

  • Tom Webster

    Thanks, Renee! I get where you are going with your example, but it’s equally arguable that no matter what generic alternative an online commenter has in mind, a vote for “ziploc bags” IS a vote for “Ziploc Bags,” in the sense that this is the dominant top-of-mind brand. Whether or not Ziploc suffers from “kleenex syndrome” is a separate, but related problem.

    And my response to your comment probably guarantees that I’ll get a crate of them from S.C. Johnson, but to date – still no Maybach.

  • http://blog.ecairn.com dominiq

    One of the other thing that bother me on sentiment is that “automatic tools” score “mentions”. If, for example, one client praised the brand in 10′s of his/her tweets and then follow up with a “These guys are BS”, the last negative comment should erase anything that was said by this person right ?

    Nevertheless, automatic “anything” is a very wrong and risky approach to social media. I wrote about sentiment analysis in the past, http://blog.jenniferlindsay.com/2010/07/06/why-sentiment-analysis-cant-work-and-why-it%E2%80%99s-a-damned-good-thing/, feedback are warmly welcome !

  • http://www.webmalama.com Aaron Brown

    I think there is value in measuring every mention because regardless of if I prompted it (or someone else prompted it), and regardless of if the mention is completely off-topic, it’s still awareness of my brand/product…sort of the old PR axiom, “Any press is good press.”

    At the same time, I also see value in smarter data aggregating where we can get a better sense of which things to measure and which to ignore. In an ideal world, of course I’d like to know which mentions have the most influence toward whatever my end goal is.

    By the way, I’ve posted a link to this article and opened comments on it at the LinkedIn Social Media Marketing Research & Analytics subgroup. Please join in the conversation there as well if you’re so inclined.

  • http://researchgeek.wordpress.com AJ

    Tom – really interesting post, something I’d never previously thought off but in retrospect seems blindingly obvious and essential (like a Jimmy Page Guitar riff).

    In fact, I wonder if aided and unaided even goes far enough in social media research. If I’m scanning blogs and discussions repeated mentions of my brand in the comments, whereby we’re just using the brand as an example of a concept, may not be that useful at all. Is this thread going to do Ziploc bags any good? Whereas if this were a comment thread on a food site where somebody had asked about storage it would have much greater value.

    Although, having said that, I don’t think I’d ever heard of Maybach before this post, so they can chalk up a new prompted recaller.

  • Tom Webster

    Make sure when you buy that Maybach, AJ, you use my coupon code: theydontusecouponcodes

Previous post:

Next post: