## Which 9 Lord Ashcroft constituency polls are wrong?

by Stephen Tall on March 6, 2015

So far, Lord Ashcroft’s £million-plus polling spree has brought us 177 individual constituency polls.

Which leads me to my question: which 9 of them are wrong?

The reason why I ask that is simple. Reputable opinion polls like Ashcroft’s are accurate 95% of the time to within a margin of error of +/-3%.

The mirror of that is that 1 poll in every 20 will be outside the margin of error: to all intents and purposes it’s wrong. It may be wrong in a direction which makes the seat even safer for the party shown in the lead; or it might flip the result entirely.

So, out of those 177 Ashcroft polls, it’s likely that around 9 are giving a misleading snapshot of the state of the race.*

Now, in national polling, this doesn’t matter much — there are so many polls published the fact that 1-in-20 polls are ‘rogue’ is insignificant.

But in constituency polls, where Ashcroft’s is the single data point available, it does start to matter. Indeed it could become self-prophesying if the public is continually told that Party X is certain to win/lose.

None of this is intended as snark against Lord Ashcroft by the way. He has been exemplarily transparent in publishing his polls, and has always made clear they are snapshots, not predictions.

But the simple fact remains: 9 of them will have been wrong and we have no idea which 9 they are.

* I can’t rule out the possibility that all 177 have bucked the statistical trend and are completely accurate. Or, indeed, that a lot more are wrong.

## 3 comments

Stephen – just as you can't rule out they're all accurate, you can't rule out that 18 may be wrong as well 😉

by Simon Foster on March 6, 2015 at 2:49 pm. #

I get the premise behind your post but I’m afraid statistical sampling theory doesn’t quite work like that. It’s a lot, lot more complicated.

I could write an even longer page on this but I’ll try to summarise the key points:

* It is not true to say 1 in 20 polls will be ‘wrong’

* All the survey estimates have an estimated margin of error of, supposedly, +/- 3%

* However, this is only true where the proportions are around 50%. Where the proportions are lower, the margin of error decreases

* However, this would only be the case where the sample is 1,000. In practice, the proportions are based on those expressing an intention to vote. So maybe 600-700. So the margin of error is greater

* The fact that all polls are weighted also makes the margins of error wider

* the cumulative effect of all of this is that I’d estiamte the true margin of error may be nearer +/- 5% for most telephone polls of sample 1,000

* That means that the true value has a 19 out of 20 chance of being within +/-5% of the published voting intention figure

* (For an abstract concept such as a voting intention, I’m not sure that a ‘true value’ has any meaning anyway.)

* Strictly speaking, this only applies to telephone polls. Online polls are typically drawn from online panels, so normal sampling theory doesn’t apply. If anyone quotes a margin of error from an online poll, it is pure guesswork

So, in short, it doesn’t make any sense to talk of any given poll being ‘wrong’ as such. What you could say is that of all the voting intention figures in all the polls, around 1 in 20 are probably more than 5% out. (Even that is an over-simplification.)

Not sure if this post is more confusing than enlightening, but I’ve tried!

by Karsten Shaw on March 6, 2015 at 8:42 pm. #

I would guess within these other qualifications that it is between a range around the 9 (or whatever) there is a chance of the being individually out of range. So perhaps there is a 95% chance that all but 6-12 polls are within a 3(etc)% range of accuracy. Numbers are examples as my 1981 A level Maths does not qualify me to answer this numerically.

by erlend on March 8, 2015 at 2:00 pm. #