by Stephen Tall on November 8, 2012
For those who don’t know Nate Silver, he’s the analytical guru behind the FiveThirtyEight blog (named after the number of electoral college votes), now housed at the New York Times, which scrutinises and filters opinion polls. He first rose to prominence four years ago after predicting the winner in 49 of the 50 US states during the Obama-McCain presidential contest, and re-inforced his reputation two years later by forecasting correctly all 35 senate races.
His cred has now reached near-messianic status. First, because yet again his methodology has been validated — if Florida is eventually declared for Obama, Nate will have another clean sweep of predictions. Secondly, because he has consistently called-out the media for portraying the race as tied despite the high likelihood that Obama has maintained a solid electoral college majority throughout the year. And thirdly, because of the personal insults that have been flung in his direction over the past few weeks by Romney supporters aggrieved that Nate didn’t have better news for their guy.
I’m a huge fan of Nate Silver’s forensic approach and FiveThirtyEight has been my first port of call to find out how the US presidential race is actually going, rather than through the mainstream media’s unfocused lens. But Daniel Engber makes a fair comment on Slate.com when he points out: “Nate Silver didn’t nail it; the pollsters did.”
And that is, I suspect, the single biggest reason this country hasn’t yet, and is unlikely to, produce our own version of Nate Silver — because we don’t have the necessary quality of polls and because we don’t have a media interested in actually finding out what’s going to happen. Let me unwrap those two assertions:
1) Absence of local polling
The reason most commentators could get away with their blather that the US presidential race was tied was because they reported the headline national numbers. These did indeed show Obama and Romney to be essentially neck-and-neck, and that’s how the national result has finished. But averages conceal as much as they reveal. What Nate Silver (and others psephologists such as RealClearPolitics and Talking Point Memo) highlighted was the data from state polls, in particular the ‘toss-up’ states which would decide the next occupant of the White House.
The same mistake that Nate Silver’s decriers made is almost universally made day-in-day-out by British pundits using national polls to project the next general election result. Ironically this includes many commentators who’ve praised Nate Silver’s approach in the US and yet disregard its applicability in the UK (yes, I’m looking at you John Rentoul).
Everyone knows that results differ across the UK: that what happens in Cornwall won’t be repeated in London or Scotland or the East Midlands etc. Yet when it comes to projecting the results of the next general election, the media is all too happy to shove a load of polls in the blender and hope it approximates to reality.
The problem is highlighted by Nate Silver himself. He attempted to devise a model for forecasting the last UK general election based on the polls. The final projection showed the Lib Dems in second place and winning 120 seats. Does this show Nate’s voodoo is a load of hooey? No, it shows that the British focus on national polling is a poor basis for projections of numbers of MPs. And it is also fair to point out that Nate Silver 1) inserted many caveats into his prediction, and 2) came up with a range of projections, the final one of which (scroll down to the bottom of this page) wasn’t so very far off.
2) Absence of media interest in accuracy
So if we accept that national polling is a deeply imperfect guide to projecting what the House of Commons will look like after the next election, it’s fair to ask the question why doesn’t the media — why don’t the polling companies — commission the kind of research that is needed to make accurate forecasts? It’s a question easily answered with another question: what’s in it for them?
I’m sure the polling companies themselves would be very happy to undertake more regional polling, to find out if the results in the North-West will differ from those in East Anglia. But they’ll need paying, not least because political opinion polling is only a very small part of their business (albeit the most publicly visible). And as there isn’t a strong, profitable regional media it doesn’t happen. There have been sporadic attempts to do it: PoliticsHome and The Sun have both commissioned extensive polls of marginal seats, but these are special one-offs so no reliable trend emerges that allows a Nate Silver-style calibration.
The blunt reality is that the news media craves excitement more than it hungers for truth. It is much cheaper and easier to commission a monthly survey and then inflate the results way beyond what the data should allow. We’ve all seen the kinds of headlines newspapers revel in — ‘Poll blow to Tories as support plunges 1%’, ‘Labour to win 100+ majority says latest exclusive poll’ — and yet journalists continue to write them even though they know deep down how flimsy the evidence is.
I’ve read lots of adulation of Nate Silver in the British media in the past 24 hours. I wonder if any of those journalists who’ve penned those articles have thought, even for a moment: I wish I had the confidence to write about polls with the same kind of rigour he does. I’m not holding my breath.