We know some of the statistics deployed in the EU referendum campaign were at best dubious. But how did both sides deploy stats to make their points? Nambassa Nakatudde takes a look at the hundreds of campaign leaflets donated to the LSE Library archive, and finds a number of alarming traits: an almost total failure to source statistical claims, a tendency to draw invalid comparisons and selective infographics that left out key information.
Millions of leaflets were distributed in the run-up to the EU referendum. Throughout the campaign, and indeed after it, there have been many attempts to factcheck the statistics quoted by campaigners: the BBC, InFact, FullFact and Channel 4 were among the organisations that tried to verify the figures. This post makes no such attempt. Instead, I want to take an objective look at the way statistics were deployed in campaign materials.
A full list of each of the organisations from which at least one leaflet has been analysed can be found at the end of this article. All material was received by post or on the street and subsequently donated to the LSE Library Archives during the campaign period (20 February – 23 June 2016). For reasons of confidentiality, the addresses which received the discussed campaign material cannot be disclosed. However, leaflets were received in a wide range of UK areas, including but not limited to Cambridge, Gloucestershire, London, Northamptonshire and parts of Wales.
Few organisations did not use statistics in at least one piece of campaign material. Those that chose not to included the Remain-leaning Party of Wales and Greens for Europe. Some material distributed by the Leave campaign groups Conservatives for Britain, Green Leaves, Cross Party, Lexit and several trade unions did not contain statistical information either.
Remarkably, one completely even-handed piece of campaign material was donated to the LSE Library Archives – a booklet distributed by Intercross Britain. All the other donated campaign material firmly supported either a vote to remain in the EU or a vote to Leave. The message obviously varied, but the techniques used by campaigners on both sides were for the most part identical. Loosely defining a statistic as a numerical fact or figure used to convey a message, a look at the letters and leaflets with an analytical eye raises three key questions.
1. Where has this come from? Missing background and source material
Usually, campaigners neglected to include a source and/or date for figures. It is overly optimistic to assume that the average voter would go to the trouble of factchecking quoted statistics, but completely excluding sources from campaign material means the more inquisitive voter will struggle to find their source. Furthermore, it gives organisations license to publish without much fear of retribution for any misinterpretation of data – only the most eager readers would go to the pain of contacting an organisation for data sources.
Perhaps the best-known unsourced statistic was the £350m figure, described as savings which could be invested in the NHS following the UK’s departure from the EU. Grassroots Out, Vote Leave and Better Off Out all used this statistic in the LSE Library’s archived material. While none of the organisations quote a source for the figure, perhaps more importantly none give a description of the methods we now know were used to arrive at the statistic.
Maybe including such detailed information doesn’t make for the most attractive of headlines? Still, simply a link to a file detailing the methodology, and in an ideal world including source data would suffice. There is another problem with the £350m figure because it represents the gross, rather than net contribution – a piece of information which is not stated.
However, in one instance a leaflet did source information given in other campaign material. This, a piece distributed by UKIP, was the only material to employ statistics to debunk claims made by another campaign (please see Figure 1). Sadly, it must be noted that these are the only sourced statistics in the leaflet.
However, to their credit, a few organisations on both sides of the debate consistently included the sources of their statistics. These included the Liberal Democrats supporting Remain, and on occasion Better Off Out campaigning to Leave.
2. Apples and oranges, and everything in between
On some occasions, statistical comparisons equated apples with oranges. For example, Figure 2 shows a section of a leaflet distributed by Grassroots Out. It makes a comparison between infrastructure spending and the UK’s contribution to the EU as an international institution. Although each international institution serves a distinct purpose and so is not directly comparable, a better comparison might be one between UK spending on the EU and UK spending on another international organisation such as NATO or the UN.
What do I mean by ‘everything in between’ the ‘apples and oranges’? These things are statistics which when left out change the impact of material, and facilitate selective reporting. Figure 3 is an example. It can sensibly be assumed that the vast majority of voters do not know the UK youth unemployment rate off the top of their head. Given this, including the UK figure in the chart could help the reader to contextualise the information. In Figure 4 I show firstly how the chart could have included the UK, and then all 28 EU member states – not just those with some of the highest unemployment rates in the EU (in white are those countries which are included in the original leaflet chart).
3. Infographics, charts and startling use of colour
They say a picture is worth a thousand words and given the indisputable efficacy of statistics, an infographic is perhaps the most potent combination of the two. Both camps used charts and infographics as a means of conveying statistics.
In each of the materials examined there were very few instances (with the exception of titles and headings) where bold or coloured information did not include at least one statistic. Indeed, numbers often featured as the centrepiece or headline of material, in addition to making an appearance in the body of the text (Figure 5).
Statistics feature in almost all of the reviewed campaign material. This is perhaps for no other reason than the fact that campaigners are aware of their effectiveness. Sometimes even the simplest of practices which could allow for voters to make better sense of statistical information have been neglected. If we assume that this is not because of a lack of statistical knowledge or inadequate resources, the inference must be that campaigners simply did not want their claims to be scrutinised. The misuse of statistics cast an alarming shadow over the referendum campaign, and the consequences may be felt for many years.
At least one leaflet or letter from each of the following organisations was analysed: Vote Leave, The United Kingdom Independence Party, Better off Out, Conservatives for Britain, Grassroots Out, The Bruges Group, Leave.EU, The EU Out Post, Green Leaves, TUA EU Trade Unionists Against EU, ASLEF & RMT & BFAWU, The Cross-Party Campaign for an Independent Britain, Lexit the Left Leave Campaign, Inter Cross for Britain, Universities for Europe, HM Government, The Conservative Party, The Liberal Democrat Party, The Party of Wales, The Labour Party, Greens for Europe, Britain Stronger in Europe.
The 2016 EU leaflets are held at LSE Library and form part of a unique range of archives that document the relationship between Britain and the European Union. To find out more see here or contact the curator for politics.
This post represents the views of the author and not those of the Brexit blog, nor the LSE.
Nambassa Nakatudde is studying for an MSc in Statistics at the LSE. She has worked for the UK Parliament in the House of Commons Library as an Assistant Statistician.