Tom Webster, writing and speaking

Customer Service, Survivor Bias, and Missing Bombers

Added on by Tom Webster.

Mt Wilhelm US PlaneThe rise of real-time social media management tools has given companies an enormous capability to address consumer issues, and the ability to potentially resolve those issues in the light of day for other customers and prospects to see. In that sense, social media management suites, dashboards and command centers are the ultimate ways to spot the "squeaky wheels," and apply grease as needed. I heard recently of a well-known brand that has taken to distributing a daily one-sheet summarizing some of these issues to internal stakeholders. This daily "trouble sheet" is probably seen as a valuable shorthand resource for busy managers to get a sense of where problems may lie with their products, and certainly this fire-fighting aspect of social media management is a crucial function for customer service and maintaining relationships.

The potential problem with these one-sheet "fire reports," however, lies in what they don't show, and the assumptions managers could make based upon this "silent" data. I'm reminded of one of my all-time favorite tales of statistical heroism here: the work done by Abraham Wald during WWII to improve the survivability of Navy bombers.

Wald's work has been reprinted and analyzed by the Navy over the years (it's here if you'd like the source material) but here's the gist of what he did. A bomber is a big, slow-moving target--relatively easy to shoot down, compared to a fighter plane--so the Navy wanted to investigate ways they could add armor to the fuselage of their bombers to improve survivability. The tradeoff with armor, of course, is weight: they couldn't simply armor-plate the whole thing, or it wouldn't get off the ground.

Wald examined the damage on all the bombers that had returned, and found that some sections of these bombers were frequently heavily damaged, while other sections were relatively damage-free. His recommendation? He advised the Navy to apply armor to the least damaged areas.

Would you have made that leap?

Here's what Wald figured out: an anti-aircraft gun on the ground blasting away at a bomber isn't really aiming selectively; it's just blasting away, and wherever it hits on a bomber is as much a function of luck as it is skill. If you buy that, then you should be able to buy that the distribution of hits on a bomber should basically be random. But that isn't what Wald discovered.

If you assume that hits are random, and you have an assortment of bombers that are heavily damaged in some areas, but lightly-or-not-at-all damaged in others, there is only one explanation: the bombers that were hit in the latter areas didn't return. The bombers that were heavily damaged made it back, so obviously they can take a few hits in those areas and still fly. Wald correctly predicted the "silent evidence": the existence of bombers that had to have been hit in the lightly-damaged areas and didn't make it back. The areas that looked "the strongest" amongst the surviving planes, therefore, were actually the weakest. Wald's discovery of this survivor bias led to the reinforcement of those areas, and the return of more bombers (and pilots) back home.

When we fight fires in social media, and make changes to our product or service offerings based upon the "hits" that we take, we are essentially reinforcing the most heavily damaged areas--and again, there is a customer service and retention function for that. The problem is when we take the same data and use it for R&D or strategic purposes, and ignore the potential silent evidence--the opinions of consumers who gave up quickly, didn't give your product a chance, or lacked the patience, will or temperament to air their grievances publicly.

The good news here is that this silent evidence is eminently findable, using primary research (this is one of the main reasons we developed The Social Habit) a smarter deep dive into your longitudinal social media data, and/or correlating your return/exchange data with your social data. In any case, it's all part of what I like to call doing the work.

Or, in the words of Erich Hartmann, "fly with your head, not your muscles."