We should take all polls, no matter how well constructed, with a big pinch of salt, writes Nick Anstead. Here, he lists five unknowns as to why this is especially the case for the Scottish referendum.
Another day, another poll appears for the Scottish referendum. On Friday, an ICM poll with the Guardian, and – among those who have made their mind up – put ‘No’ at 51 per cent and ‘Yes’ at 49 per cent. 17 per cent of those polled remain undecided. The past week of the campaign has been notable for the huge role played by polls in driving both the news and the political agenda. Indeed, the ‘Devo-Max’ offer from the three major parties at the beginning of last week largely seemed to occur because a YouGov poll put the ‘Yes’ camp in the lead for the first time.
I have recently been writing on the idea of public opinion, and one really interesting thing that comes across from the literature is the tension between polling as a science and an art. As Susan Herbst details in her book Numbered Voices, an account of the early days of the modern opinion polling industry in the United States, one of the great rhetorical innovations by George Gallup and his contemporaries was arguing that public opinion could be measured in a scientific way, certainly in comparison with older methods such as straw polls. But the truth is that, no matter how rigorous the method, opinion polling has always required a healthy dose of creative thinking and skillful judgement.
This truth is especially evident in the case of the referendum, as there are so many factors which might have an impact on the final result. Going into the last few days of the campaign, I would list five unknowns that mean we should take all polls, no matter how well constructed, with a big pinch of salt:
- What does “don’t know” actually mean? Journalist Dan Hodges was quick to tweet after the release of Friday’s poll that “don’t know” was a euphemism for “no”. The theory here seems to be quite close the shy Tory factor or Bradley effect, namely that people have already made up their mind on voting ‘No’, but are not willing to publicly admit it. This may or may not be true, but it certainly seems that – even if people are genuinely undecided – then they might make their mind up in a predictable way, which would seem to make it more likely they would support the status quo.
- Modelling the electorate is fiendishly difficult. General elections are relatively easy to model, as pollsters have a wealth of data on previous contests. When an interviewee says they are “quite likely” to vote, for example, then how that is understood is based on a range of pre-existing polling and turnout data. But the referendum is a unique case. Partially, this is because it is asking an unprecedented question. In addition, allowing 16-17 year olds the vote adds a completely new cohort of would-be voters. But perhaps most importantly, the 97 per cent registration rate reported earlier this week is completely unprecedented. This means that many more members of the public will be eligible to go to the polling booths on the day of the vote than has ever previously been the case (whether they do or not is a very different matter. See point 4 below).
- What impact could postal voting have? Sky News is already reporting stories about postal voters who are “regretting their choice”. Nearly 800,000 people will vote by post. This has a couple of practical ramifications. The first is that postal voters, obviously, will be immune from events in the last days of the campaign. The second is that postal voting of this kind presents a methodological issue for pollsters. Polls are a snapshot of public opinion at the time they are taken, so we would assume that the polls immediately before the referendum will offer the most accurate predictions. However, when nearly 20 per cent of the electorate have already cast their vote, it means that the same poll might not accurately reflect their votes.
- Who will be able to get their vote out? A lot has been made of the Yes campaigns grass roots mobilisation, as distinct from the more traditional top-down approach of Better Together. These characterisations are probably a bit glib, but there is no doubt that, while Better Together can fall back on the organisational muscle of the Labour Party, the ‘Yes’ campaign has linked itself with a broad range of civic and political groups. Given the very high registration rate among voters and the seeming closeness of the race, effective get out the vote efforts from either side might carry the day.
- How do we understand Labour supporters moving into the ‘Yes’ camp? Slightly less of a polling issue this one, but one really interesting element of the referendum campaign thus far has been the number of Labour supporters who are moving into the ‘Yes’ camp (as Peter Kellner notes in his commentary on the recent YouGov poll that put ‘Yes’ in the lead). There are interesting parallels here with Rob Ford and Matthew Goodwin’s important revisionist work on UKIP, where they argue that a significant part of UKIP electoral support is coming from former and natural Labour voters. Ford and Goodwin’s argument is that this UKIP-supporting group feels estranged from the modern Labour Party, Westminster politics and deeply economically insecure – many of the same characteristics that are driving the movement towards the ‘Yes’ camp in Scotland.
Of course, it may be that, come next Friday, the pollsters have got it dead right. It may also be that a last minute swing to one side makes these variables of academic interest only. But, in the meantime, spare a thought for the pollsters grappling with one the most difficult challenges they will have ever faced.
Note: This article was originally published on Nick Anstead’s blog and gives the views of the author, and not the position of the British Politics and Policy blog, nor of the London School of Economics. Please read our comments policy before posting. Featured image credit: Chilanga Cement CC BY-SA 2.0
Nick Anstead is a Lecturer in the Department of Media and Communication at the London School of Economics. His personal blog can be found here: http://nickanstead.com/blog/ and tweets from @NickAnstead