by Stephen Tall on November 30, 2014
What happened there, then? I refer to the LibDemVoice survey of party members reported here on Wednesday which showed Daisy Cooper with a clear lead over Sal Brinton in the contest for party president; when actual votes were counted the result was reversed. Is this moment to the LDV surveys what the 1992 election was to the pollsters? In haste, here are a few initial thoughts from me…
1. The sample itself is drawn from the 1,500+ current Lib Dems signed up to our members-only forum. This is therefore self-selecting; as is who chooses to respond. It isn’t the random sampling adjusted to be demographically representative that professional pollsters would use. But, then, nor do we claim it to be: each and every survey post makes clear where our sample is drawn from and states “we make no claims that the survey is fully representative of the Lib Dem membership as a whole”.
2. We have very few occasions to test the LibDemVoice survey sample against actual votes or elections. There have been only three previous examples: the 2008 and 2010 elections for party president, and the 2010 special conference to approve the formation of the Coalition. In each of these, the survey results accurately predicted the winner (if not always the margin). But past success is, as we can see, no guarantee of future performance!
3. The reliability or otherwise of the surveys has often been contested, though. Mark Pack produced a handy FAQs here, while, from an external perspective, YouGov’s Anthony Wells offered a fair assessment of their strengths and weaknesses here.
4. Here’s a quote from Anthony’s verdict that bears repeating today: “I do also worry about whether polls that are essentially recruited through online party-political websites or supporter networks get too many activists and not enough of the armchair members, or less political party members … All that said, while they aren’t perfect and Mark and Stephen never claim they are, I think they are a decent good straw in the wind and worth paying attention to, especially given the verification of whether respondents are party members.” Which I think remains fair comment.
5. We know and publicly state the LibDemVoice sample of party members is skewed towards activists (and male activists at that). This hasn’t mattered in the past (see point 2). Clearly it did this time – as I highlighted might be the case in my heavily caveatted write-up of the results. Actually I was relieved to re-read my post again last night. Why? Because I wouldn’t change a word of it with hindsight: “our results below need to be taken with a pinch of salt … in an internal election where personality is a key factor I can’t be confident that our surveys are necessarily reliable measures … Congratulations to [Daisy] on this strong showing. However, I stress the caveat already inserted that our surveys are skewed towards activist members who may well not be representative of the wider ‘armchair’ membership.”
6. I’ve had a quick dig into the results. There’s nothing obviously different about the data compared to previous surveys. Men out-number women 4:1, which is nothing like the party membership mix but is (unfortunately) standard in our surveys. We do much better on the mix of ages (55% over 50, 45% under) and geographies (though we’re under-weight in the south-west). However, one stat may be key: almost half those who respond are conference representatives (and of course the other half are keen enough party members to have registered with our members-only forum).
7. My best guess of what happened, then, is that the survey reasonably accurately estimated Daisy’s support among party activists. However, the sample was clearly under-weight in non-activist (‘armchair’ members) and on this occasion that made a big difference. Especially relevant to this, I suspect, was the impact of high-profile endorsements for Sal Brinton from Paddy Ashdown and Shirley Williams, probably the two most popular and influential figures in the party, in the leaflets / emails sent to all members. Given none of the three candidates was a household name current MP, the recommendation of those two will likely have made a major impact on those members least likely to complete an LDV survey.
8. Because the survey got this one wrong, does that mean there’s no purpose to LDV’s surveys in the future? Well, I’m biased clearly — so feel free to ignore / dispute my views on this! — but I don’t think so. That doesn’t mean there aren’t lessons to be learned (there are, but that’s for another post, another day). But here are three reasons I think the surveys continue to be a worthwhile part of what this site does.
Just make sure you keep that pinch of salt handy…
* Stephen Tall is Co-Editor of Liberal Democrat Voice, and editor of the 2013 publication, The Coalition and Beyond: Liberal Reforms for the Decade Ahead. He is also a Research Associate for the liberal think-tank CentreForum and writes at his own site, The Collected Stephen Tall.