Was YouGov’s leadership poll “technically flawed”? (UPDATED)

by Stephen Tall on December 4, 2007

That’s the suggestion that was made last night in a comment on Lib Dem Voice by Martin Tod, the party’s prospective Parliamentary candidate for Winchester, who noted:

Just had a quick look and there’s an important technical flaw with the poll… The poll sample’s age split isn’t even close to the party membership’s actual age split. It (unfortunately) hugely overstates young voters and understates those over 60.

If my reading of the data is right, YouGov’s weighted sample of 678 Lib Dem members comprised:

* 237 (35%) 18-39 year-olds
* 217 (32%) 40-59 year-olds
* 224 (33%) 60+ year-olds

I don’t know how this corresponds to the overall party membership; but my guess, if only from observation, is that Martin is right, and that the weighted YouGov sample is indeed hugely overstating the proportion of members under the age of 40. Which candidate this might have most benefited – or disbenefited – is simply conjecture. But perhaps the results of the poll should be treated with great caution until this point can be clarified.

UPDATE:

Peter Kellner has kindly provided this response:

We derived our age profile by comparing the profile of Lib Dem members in our panel with the demographic profile of our overall panel. This gave us a median age of 51. Given that there may have been changes since the Seyd/Whiteley survey which was, I believe, some years ago [and gave an average age of 59 for Lib Dem members], we were comfortable with this.

However, my calculation is that were we to have weighted the data using a median age of 59, this would give us Clegg 55-Huhne 45, instead of 56-44.

Given the margin of error that is always liable to occur in a sample of 678, this difference seems too slight to cause huge concern. The main point remains the one that we made to Sky News when delivering the results: given the large number of don’t knows among those who had not yet voted, the result could be anywhere between a comfortable Clegg victory and a very tight finish with the contest going either way.

Enjoy reading this? Please like and share:

No comments

It’s presumably not entirely conjecture as to what the result would be – we can rebalance the scores according to the right age ratios and see what the impact is.

I think that Clegg won over the under 35s (which corresponds from anecdotal feedback on the LDYS website) which means that, if the YouGov poll is otherwise correct, the contest is going to be a lot closer.

I was actually wondering a about the ABC1 v C2DE split – that’s ratio has to be right as well, but I doubt that YouGov have access to that kind of information (the latest public work I’ve seen was the “Neither Left nor Right” book from a few years ago.

by Peter Bancroft on December 4, 2007 at 9:52 am. Reply #

This is not necessarily a technical flaw. The question of whether to stratify your sample, and which strata to use is a difficult one. Get it wrong and you make the result less accurate not more.

YouGov may, for instance, have a good reason to stratify by, say, sex and region and not age. But they do know the ages, so they can stratify and weight by age if that seems to be a good idea. Do we actually know whether they did or not?

I share, of course, the common Lib Dem reservations about YouGov’s methods in general, but I wouldn’t leap to the conclusion that they went wrong any extra this time.

by Joe Otten on December 4, 2007 at 9:54 am. Reply #

OR – we could wait until December 17th

by Cheltenham Robin on December 4, 2007 at 10:06 am. Reply #

Isn’t every poll by definition “technically flawed” and needing to be treated “with great caution”. The mere fact that James Graham (plus several other bloggers) was one of those polled speaks volumes. I am a great fan of the Jamester, but he is hardly representative of the Mrs Muggins who makes up much of our membership.

by Paul Walter on December 4, 2007 at 10:13 am. Reply #

Last time YouGov overstated Hughes and understated Campbell. They weren’t far off on Huhne’s percentage but wrongly had him ahead because of understating Campbell.

At the time I didn’t believe their poll because I thought those in the YouGov panel were unlikely to be representative, and were likely to be inclined towards the people who were swining Huhne’s way.

It is very possible that the same factors could skew the poll Clegg’s way this time.

by Sam on December 4, 2007 at 10:28 am. Reply #

I think Cheltenham Robin has a very good point and once you try unpicking things like this (because it may or may not concur with your own view) you are in danger of getting completely tangled up into more and more meaningless analysis.

However, I’m going to ignore that warning and wonder what the gender split and impact of the poll was?

by Jo Christie-Smith on December 4, 2007 at 10:45 am. Reply #

Here is th =e opinion of a former senior pollster, now retired and teaching Politics:

‘Clegg likely to win as many votes will already have been cast. Women twice as indecisive as men. Clegg wins because seen as appealing, not policy or competence. Clegg the one to deal with Cameron. (Can people tell them apart?). I think yes, as members clearly want LDs to be a left wing, no difference perceived on left/right positions of Clegg/Huhne.’

by Chris Squire on December 4, 2007 at 12:05 pm. Reply #

Cheltenham Robin and Jo have a point but I really would like to know more about the validity of the methodology behind this poll.

The headline figure might turn out to be right for the wrong reasons if you see what I mean.

If it really does tell us valid information about Lib Dem members (beyond who they are going to vote for for leader) then it could be really useful.

by Ed on December 4, 2007 at 12:28 pm. Reply #

One factor here is that these are self-declared ‘party members’ (no-one had to show a party number to participate). The poll first asked how respondents would vote in a General Election ‘Tomorrow’ and then whether they were members of any political party (list provided). That is a crude filter for mischief makers perhaps…

by Edis on December 4, 2007 at 12:53 pm. Reply #

I’ll treat this YouGov poll as I do all of them – with contempt. They’re voodoo polls – not based on any statistically valid methodologies and when Kellner talks guff about MoE – he knows its spin as MoE can only apply to randomly sampled polls. YouGov’s panel poll is self selected, not randomly sampled and the subset of confirmed Lib Dem members likely to be very small in total.

I’d guess the reason for the sample size of 648 is that it represents pretty much everyone on their database who ticked the ‘I’m a Lib Dem member box’.

by Dan Falchikov on December 4, 2007 at 12:53 pm. Reply #

“Women twice as indecisive as men.”

Chris Squire – this “former senior pollster, now retired and teaching Politics” – it isn’t the “Pub Landlord” is it?

by Anonymous on December 4, 2007 at 2:21 pm. Reply #

Sorry – was 11!

by Paul Walter on December 4, 2007 at 2:22 pm. Reply #

Peter seems to have cleared up Martin’s point: the age profile could have mattered but doesn’t. Obviously it is possible that the whole You Gov panel is warped by non-age related issues (self-selection) but we don’t have any evidence to that effect. In addition, neither camp is claiming that it is wildly wrong, so I think it is our best guess of where things are at the moment.

by tim leunig on December 5, 2007 at 1:22 am. Reply #

13 – or not.

YouGov has an appalling record of predicting Lib Dem internal elections and the party’s public share. Almost as bad as the Parliamentary Party…

I suspect Clegg is in the lead – but there are loads of people who voted for Ming who are now backing Huhne – mainly due to Clegg’s appalling campaign.

by Dan Falchikov on December 5, 2007 at 1:55 am. Reply #

Leave your comment

Required.

Required. Not published.

If you have one.