Tom Webster, writing and speaking

Two Cents on Jason Calacanis's comScore Imbroglio

Added on by Tom Webster.

Jason Calacanis is angry at comScore and would like us all to boycott their measurement service. The gist of his ire is that comScore is now augmenting their panel-based measurement service with a server-based measure which requires sites to install a small pixel-tracker across their pages/widgets/etc. Since comScore is charging $5000 to have this tracker installed, Calacanis claims that comScore is blackmailing sites--pay up if you want the "accurate" numbers. He goes on to berate comScore for being wildly inaccurate, especially compared to server-based measurement platforms like Quantcast or Google. I don't have a dog in the comScore fight, but I do know that raw server-based measurement data is also wildly inaccurate, but because it inflates (rather than deflates) unique user counts, no one complains. This reminds me of the Clinton/Dole race of 1996, where so many of the pre-election polls called for a Clinton win, but by a significantly larger margin than his actual margin of victory. Since a win is a win, the polls largely got a pass from the American people--but the polling was pretty off, regardless.

My sense is that most people wouldn't complain if their server-based measures overstated unique users by a couple hundred percent or so--which some experts believe cookie deletion alone could cause--so comScore naturally gets a little more scrutiny when their panel-based measures appear to understate users by comparison. But it really doesn't matter what site owners or publishers want--it's what advertisers want. Generally, they want accuracy, and they want qualitative / demographic / psychographic / geographic data as well. The only source for the latter, of course, is survey research, which comScore access via its panel.

Survey vs. Server as false choice continues to be one of my big themes here at BrandSavant, and the answer should always be both. Server-based measures give you perfect measures of meaningless numbers ("Traffic") while survey-based measures give you imperfect measures of meaningful numbers. Both are estimates.

So, you could look at what comScore is doing the same way Calacanis is--holding us ransom for "the real numbers"--or you could see this as comScore trying to make a better product, and that product is going to cost more money. Server-based measurement companies are quick to say "I told you so" to comScore's move to mimic their cookie-tracked data, but the real story here is that comScore's hybrid approach signals an improved approach, and the cookie-trackers of the world would be wise to incorporate survey/panel data into their own products, even at the expense of millions of phantom "users."

I believe in the hybrid approach in theory. I can't comment on comScore's actual product, so I won't. I do think the price to play is quite high, and will be potentially onerous for some startups and small businesses. I assume the market will take care of that. So, at least in that respect, if you think comScore's price is too high then by all means "boycott." It'll probably come down if they can't sell it. But claiming that they are holding back the "more accurate numbers" assumes facts not in evidence. Panel-based measures are not less accurate than server-based measures, merely different. The ideal solution would be to focus on continuous improvement of server-based measures by calibrating them with survey-based measures and vice-versa. Pitting them against each other, as Calacanis and many of the commenters to his article do, is drawing a line in the sand where none should exist.