LSE - Small Logo
LSE - Small Logo

Ros Taylor

February 22nd, 2016

Polls apart: why we need to treat all EU referendum polling with caution

7 comments

Estimated reading time: 5 minutes

Ros Taylor

February 22nd, 2016

Polls apart: why we need to treat all EU referendum polling with caution

7 comments

Estimated reading time: 5 minutes

goodwin

Why do phone and online polls diverge so greatly? Matthew Goodwin discusses why the figures for Leave and Remain differ according to the polling method and argues that we need to approach results with caution. He also looks at how important Boris Johnson’s intervention will be.

Game on! Britain’s referendum on its continuing EU membership is scheduled for June 23. So how are the Remain and Leave camps doing – what is the state of the race so far?

Well, it might be crunch time but there is also considerable confusion in the polls. Let’s take a step back and look at what is actually going on below the surface. As I wrote for The Times Red Box this week the picture is far from clear. The headline, take-home point is that the polls conducted online tell a very different story about what is likely to happen to the polls conducted via telephone.

For this bulletin, I ran the numbers on all polls conducted since the start of December 2015. Look at the chart below. In online polls, by companies like ICM or YouGov, the referendum race looks very close. It is a dead heat. Remain is on 41%, Leave is on 41% and undecided voters represent a not insignificant 18%. If you look only at these numbers then you would conclude that there is everything to play for.

goodwin1

But now look at the average result from polls conducted over the telephone. In these polls, conducted by firms like Ipsos-MORI and ComRes, it is a very different story. Remain, on 56%, holds a striking 18 point lead. It looks set for a comfortable victory (this was also true in a new and post-EU deal Survation telephone poll that was released last night that put Remain on 48%, Leave on 33% and Undecided on 19%). It is not even close. In the telephone polls there are also a much smaller number of undecided voters, a point I will come back to.

This is not just a nerdy point. It really matters. The polls will strongly influence the referendum – the campaign, voters and journalists. Just look back to that YouGov poll a few weeks ago that gave Leave a 9-point lead. Brexit loomed. Cameron was toast. But only a few days later along came a telephone poll that gave Remain a 8-point lead, then another that gave Remain an 18-point lead, and then the one last night that put Remain 15 points ahead.

boris johnson david cameron
Happier times … David Cameron and Boris Johnson visit a London cab factory together in 2015. Photo: No 10 via a Creative Commons 2.0 licence

So what is going on?

There are different theories. One possibility is that the polls are being influenced by something called ‘social desirability bias’ – which leads some voters to give pollsters an answer that they think is socially desirable (even if this is not their actual view!) People might not have a problem admitting their desire for Brexit online, but they might be more hesitant when there is another person on the end of the telephone. This might be contributing to the strong lead for Remain in the telephone polls and the fact that these polls report a lower number of undecided voters. Who wants to admit that they don’t have a view?

There is evidence to support this. One study in North America, for example, found that telephone respondents are often more suspicious about interviews and, crucially, more likely to present themselves in socially desirable ways. Another experiment that compared telephone and internet methods found that phone surveys generated more random error and higher rates of social desirability bias. If this is true then it might be that Remain’s strong lead in telephone polls might not be as strong as we are led to believe, or it might not exist at all – although I find that one harder to believe.

To illustrate these differences, take a look at the table below where I compare results from the most recent telephone poll (Ipsos-MORI) with a recent online poll (Survation). You can see immediately how support for Remain is much higher across the board in the telephone poll whereas in the online poll by Survation there are larger numbers of undecided voters. I’ve also included results from the face-to-face British Election Study, conducted last year. In face-to-face Remain is comfortably ahead but across all three methods you can see how notable the differences are. This is why jumping on single snapshot polls is a game for fools.

goodwin2

Another very real possibility is that the polls are still not adequately representing the views of key groups in society, which could make all the difference at the EU referendum. It has already been noted how, at the 2015 general election, the samples of people in the polls were not as representative as they should have been, over-representing younger Labour voters and under-representing Conservative voters. If this problem has not been properly fixed then it could carry real implications for the referendum. Those Labour-leaning voters tend to be more positive about the EU, which might be artificially inflating support for Remain.

Another specific question is whether the polls are also accurately reflecting the average over-55 year old and young person. These two groups are at the opposite ends of the debate – the over-55s are the most likely to back Leave, while the 18-34 year olds are the most likely to back Remain. But older Britons are very difficult to reach and it might be that current polls are skewed toward more politically engaged older voters. As one pollster, Adam Drummond, recently reflected: “It is possible that the same thing as happened in the general election with young people is happening again with online polls over-representing the easiest over-65s to reach on panels who happen to be more pro-Brexit than the rest of their age group”.

If this is the case, with online samples over-representing more Brexit-minded older Britons, then alongside the evidence in telephone polls it could suggest that Remain has an edge. On June 23 the less engaged older voters who are perhaps less Eurosceptic will turn out and have their views heard. Alternatively, if the samples of older voters are accurate but more ‘normal’ younger respondents are not adequately captured then the race could be much closer than we currently think.

The solution? Well, one pollster thinks that ‘the true answer’ may lie somewhere between what the telephone and online polls are reporting. This school of thought has been influenced by the experience of Ukip, whose vote share at the 2015 general election basically fell between what the online and telephone polls were projecting. But I don’t find that explanation satisfactory – especially given the importance of the national debate and the fact that Euroscepticism is a more multi-dimensional phenomenon than support for a hard Eurosceptic party. What is clear – and is the central message of this briefing – is that until the problem is resolved we need to inject a new sense of caution into how we report the polls.

Oh, and one last thing: how important could Boris be? Take a look at the chart below, taken from a very recent Ipsos-MORI poll, and compare the reach of Boris to the only other Outers….

goodwin3

Forthcoming events

On February 24th Professor John Curtice and Matthew Goodwin will be discussing public attitudes toward the referendum at Portcullis House, House of Commons. Register here.
On February 26th Chatham House will also be hosting a breakfast briefing for journalists on the referendum, featuring Anthony Wells and Professor Sara Hobolt. Journalists can register here.

Other evidence-based blogs
Damian Chalmers asks – Ever Closer Union – Does it matter?
John Curtice writes a memo for David Cameron
Chatham House briefing paper – What drives Euroscepticism?
This post represents the views of the author and not those of the BrexitVote blog, nor of the LSE.

Matthew Goodwin is Professor of Politics in the School of Politics and International Relations at the University of Kent, and Associate Fellow at Chatham House. He is also a senior fellow at the UK in a Changing Europe project. His most recent book is the co-authored Revolt on the Right: Explaining Support for the Radical Right in Britain (2014). @GoodwinMJ

Print Friendly, PDF & Email

About the author

Ros Taylor

Posted In: Campaigns | Featured | UK politics

7 Comments

Comments are closed.