Tom Webster, writing and speaking

How To Value Customer Feedback - A Case Study

Added on by Tom Webster.

NewImageYesterday I got an invitation to take an online survey, sent to me from a leading market research company on behalf of a leading jeweler. The name of the jeweler isn't important--let's just say, they are a reassuringly expensive store that rhymes with "Spiffany." I got this survey because I've made online purchases with this retailer, and am thus on their mailing list. What struck me about this particular survey solicitation was the value it placed on my feedback: none. Sure, there was a perfunctory "thank you for taking the survey" line in there, but no actual incentive for me to take the survey. There's nothing remarkable about that fact; indeed, it's all too common for us to get survey requests that offer no incentives. It's why we associate online survey with "quick and cheap." But, as the sign above my dry cleaner says: Quality, Service or Price--Pick Any Two.

I thought a lot about this particular solicitation, and I think there are lessons here for brand marketers, market research companies, and ANYONE who solicits feedback from their customers or audience. First of all, I think it's all too common for market research vendors to helpfully offer to their clients that they don't "need" to have an incentive, because they know the numbers will come regardless. Since incentives are a pass-through, and the cost of paying survey respondents jacks up the final bill with no gain for the researcher, I think some research vendors are quick to make that part of the plan "optional," as a way of making the final proposal cost more palatable.

I'll assume, however, that the good MR companies always make it available as an option, and attempt to educate their clients about survey bias. But, for most clients with moderately large databases, you can probably get a few thousand responses to any given survey without paying an incentive.

The Three Biases

1. Who wouldn't take a survey?

Here's the problem with that as far as this particular survey is concerned. Yes, they will get responses. There are, however, three pretty significant potential non-response biases. The first, of course, is the unknowable unknown of how many Spiffany customers just wouldn't take any web survey--and what their buying habits are. This is true of any online survey, and especially true of one with no incentive. Most leading MR survey vendors will have some models of this, and will work with clients to understand and contextualize the data.

2. What are the differences between those who would, and those who wouldn't take this survey?

But here's what they almost assuredly do NOT have in this case: a differential model of survey response probability by customer segment based upon spending habits. In other words, again using our Spiffany example, how much more (or less) likely are consumers who buy $20,000 rings to take an online survey than are customers who just bought a ceramic dalmatian? (Wheel of Fortune, circa 1984 - the LAST PRIZE YOU BUY.)

I'm going to make an assumption here (and yes, it is an assumption, but the point is that you can't prove it right or wrong without doing the work): Customers who are in the top quintile of spending are less likely to take an online survey than are customers in the lower tier of spending. Or vice-versa. Or you don't know. But that's the point. If we assume the former to be true, however, then sending a survey with no incentive, no reward for completion, is likely to over-represent lower-tier purchasers.

3. Who couldn't take this survey?

Finally, it's important to remember that this was an online survey sent to the database of persons who had made a purchase at the company's website. Now, I've made two purchases with this particular retailer. The first was for about $250.00. A trinket! A mere bauble! I confidently entered my credit card information online and awaited the UPS truck. The second thing I purchased was reassuringly expensive, which required (in my case) some reassuring human contact. I made this purchase in the store, where I was not asked to provide my email address--and thus, this purchase is not connected to their online survey database.

If we again make an assumption that there is some magic price point (and it may differ by segment) over which an online purchase is ruled out, then patrons of this particular retailer who had only made purchases at this price point or higher would not have even been solicited to take the survey, while those who had made both (or only less-expensive purchases) would be.

How To Fix The Problem

The net effect of these three biases is to create a compound non-response bias that you can't even begin to model unless you've done a fair amount of work ahead of time to quantify the segments within your database, to gather data about your customers who are not in your database, and to ascertain who amongst your list is most and least likely to take an online survey. All of this work is doable. And even if you don't do this work, this sort of survey isn't worthless--after all, it will give Spiffany information about the people in their database who are in fact likely to respond to email solicitations. So there's that.

But they didn't get me. And, at a personal level, I felt devalued in a way. It wouldn't take much to show me that they truly valued my opinion. But asking me to do their work for them, for free, after dropping the chunk of change I've spent with them? The temerity! No, the more I spend with them, the more I want them to convince me that not only my custom, but also my feedback, are valuable.

In panels we have built both in the US and abroad, we've segmented potential respondents on a variety of dimensions, and we have a differential incentive tier for those segments. This at least gives us something we can tweak, so that we are as likely to get front-line retail clerks as we are CEOs. And yes, we pay the CEOs a lot more to be in the panel, or to take the survey--because we value their feedback. That's life.

In the case of Spiffany, they could have offered a differential incentive system based upon what they know about me--how much I've spent. If they don't care about big-spenders, then there's no need--but if they do, then they need to figure out a way to reward them to take the survey appropriately. Again, Quality, Service and Price: Pick Any Two.

The other thing that Spiffany needs to figure out is a way to connect my in-store purchase with my online profile. That, too, requires an appropriate valuation. "Would you like to be on our database" is going to work for some sample of Americans, but it won't for others. Why would I want to give a retail clerk my email address? Well, I can think of lots of reasons--but all of them require some valuation of that intrusion. For some brands, that's the accumulation of points, or discounts.

The Virtuous Circle

With Spiffany, however, they have to be a little more creative. They don't discount, and they don't have sales, so they can hardly offer a "VIP" program that tarnishes their halo. But, again, they do know some things about me based upon my online purchase behavior, and they can know something about me based upon my in-store behavior. How do you reward the various segments of Spiffany customers without devaluing the brand? By not using Spiffany products or purchases as a part of the reward scheme.

In other words, while you might not want to offer me a 10% discount off my next purchase, you could buy me dinner. Or a free book on my Kindle. Or access to some kind of private event. You value my segment and make me an appropriate offer. Do that, and I'll do more than let you get to first base--you might even get to third. Get to third and you've gone beyond merely providing me with an incentive to take your survey--you've strengthened my loyalty to the brand, period.

So many marketers have this received wisdom about online surveys being a great alternative because they are fast and cheap. But there is an enormous benefit to doing online surveys the right way, making an investment in all segments of your customer base, and creating a virtuous circle between the desire to share information with your brand, and the desire to spread information about your brand with others.


Lest the folks at Spiffany think I've picked on them, let me just say this: To all the fellas in the place, first: put your hands in the air, and wave them like you just don't care. Second: The blue bag. It's magic. Trust me on this one.