By analysing 121 opinion polls, Harold D. Clarke, Matthew Goodwin, and Paul Whiteley outline what happened with the EU referendum survey results. They explain why internet surveys performed substantially better than telephone ones – contrary to the post-2015 General Election ‘wisdom’ that telephone surveys should be preferred. Underlying trends showed that once methodological artefacts are controlled, Leave was almost certainly ahead of Remain over the entire last month of the campaign – and possibly throughout 2016.
After the 2015 general election, pollsters were roundly criticised for their failure to predict the Conservative victory. Did they fail again in the EU referendum? If so, just how badly did they do? Did they all get it very wrong, or did some do much better than others? And, taking artefacts introduced by polling methodology into account, what did underlying trends in Remain and Leave support look like in the run-up to voting day? Going forward, what lessons can we learn?
We address these questions by using a technique called ‘dynamic factor analyses’ on 121 internet and telephone surveys conducted between January 11 and June 22, 2016.[i] We use a statistical model originally developed in research on the performance of polls conducted in an Australian national election campaign.[ii] A large majority of the survey data employed here was gathered by nine survey organisations (‘houses’) and we also include several additional surveys in a residual ‘miscellaneous’ category for other organisations reporting one to three polls.
We begin by investigating what survey researchers call ‘mode’ effects. These effects concern the impact of differences in the way surveys are conducted on the results they produce. In the case of the 121 EU referendum surveys of interest here, all were administered either by internet (N = 81) or telephone (N = 40). Analysing the results (percentages of Remain and Leave supporters in various surveys) using our model reveals that both types of polls significantly overestimated the Remain vote share and underestimated the Leave share (see Figure 1). However, there were sizable differences between the two types of polls, with the mode effect for equaling 5.1 per cent for internet polls as compared to fully 9.2 per cent for telephone polls.[iii] Taken as a group, internet polls tended to perform considerably better than their telephone-based rivals.
What about the performance of individual survey companies? Our analysis of house effects indicates that survey firms varied markedly in their ability to gauge EU vote intentions. Figure 2 documents that, for the 9 survey organisations considered, TNS-BMRB, ICM and YouGov performed best — their overestimates of the Remain vote share were +2.24 per cent, +2.71 per cent and +3.24 per cent, respectively. Similarly, these three houses did best on estimating Leave support. TNS-BMRB underestimated the Leave share by -2.35 per cent, with ICM missing by -2.81 per cent and YouGov, by -3.34 per cent. All of the surveys conducted by TNS-BMRB (5 surveys) and YouGov (28 surveys) were done by internet, as were 23 of 26 ICM surveys.
Note: solid colored bars are for survey houses doing all or most of their polling by internet; striped bars are for survey houses doing all or most of their polling by telephone. Populus has 3 internet polls and 2 telephone polls.
The three firms performing worst were Populus, IPSOS and ComRes, with Remain house effects equaling +6.02, +.6.34 and +6.72 per cent, respectively. Populus missed the Leave percentage by -6.10 points, whereas IPSOS missed by -6.45 per cent and ComRes by -6.85 per cent. All 7 of IPSOS polls were done by telephone, as were 9 of the 11 ComRes polls, and 2 of the 5 Populus polls. Survation was another house that performed poorly (a +5.54 per cent overestimate for Remain and -5.67 per cent underestimate for Leave). All but one of Survation’s 8 polls were conducted by telephone. ORB, with a Remain house effect of +4.76 per cent and a Leave house effect of -4.87 per cent, did 9 of its 13 polls by telephone.
By controlling for mode and house effects, our analyses enable us to estimate underlying trends in the dynamics of support in EU referendum vote intentions. The results (Figure 3) indicate that Leave led Remain over the entire period from 11 January 2016 onward. The size of the Leave lead varies widely—from a low of .39 per cent (4 February) to a high of 13.2 per cent (12 June)—but Leave is always ahead.
The analysis also provides insight regarding the effects of various events that occurred during the campaign. For example, contrary to his intentions, US President Barack Obama’s widely publicized ‘UK to the back of the queue’ intervention may have boosted, rather than diminished, Leave support. In contrast, as numerous observers have speculated, the murder of Labour MP Jo Cox on 16 June may have precipitated an ongoing erosion in Leave support over the final week before the vote. That downturn in Leave’s vote intention share notwithstanding, the analysis documents that over the last month of the campaign the lower bound of the 95 per cent credible (confidence) interval for the Leave vote always exceeds the 50 per cent mark. This means that Leave was very likely ahead throughout this entire period.
What lessons can we learn?
Overall, the analyses suggest two general lessons for consumers of political polls. First, house effects associated with selected internet survey firms were quite modest, whereas firms relying heavily or exclusively on telephone surveys found themselves considerably further off the mark. After the 2015 general election, internet surveys were widely criticized and some commentators claimed that telephone surveys were a preferred alternative. At the time this assertion seemed dubious given that telephone surveys’ typically dismal response rates clearly gainsays any claim to be true ‘probability’ surveys. As documented above, the relatively poor performance of telephone surveys in the EU referendum reinforces doubts about their superiority.
A second lesson is that the ‘topline’ results from surveys are not enough. If observers had used the polling data as inputs to statistical models of the underlying dynamics of EU referendum vote intentions rather than risking being misled by the vagaries introduced by mode and house effects in successive individual polls, they might not have been surprised by the Brexit result. High quality polls by reputable survey houses should be viewed as a key resource—not a substitute—for informed political analysis.
Harold D. Clarke is Ashbel Smith Professor, School of Economic, Political and Policy Sciences, University of Texas at Dallas, and Adjunct Professor, Department of Government, University of Essex.
Matthew Goodwin is Professor of Politics and International Relations at the School of Politics and International Relations, University of Kent.
Paul Whiteley is Professor at the Department of Government, University of Essex.
[i] The polling data for the UK EU referendum are available here. “Undecideds” are allocated proportionately to the Remain and Leave categories
[ii] See Simon Jackman, ‘Pooling the Polls Over an Election Campaign‘. Australian Journal of Political Science 40: 499-517. The analysis employs Bayesian MCMC methods. Posterior distributions for model parameters are generated using Winbugs 14 in conjunction with the R2Winbugs R package.
The pooling the polls model has two equations—a measurement equation and a state equation for the latent level of Remain/Leave support. The measurement equation is:
yit = δi + αt + εit
where: yit = is a poll by survey house i at time t; δi = house or mode effect parameter; αt = latent vote intention (Remain or Leave) at time t and εit is a stochastic error ~N(0,σ12).
The state (latent transition) equation is a random walk: αt = 1.0* αt-1 + ζt where
ζt ~ N(0, σ22). εit and ζt are assumed to be independent.