With the next BBC Charter Review not too far off, public and political opinion about the broadcaster is going to be increasingly important. The recent CPS report “proving” bias at the BBC got substantial media exposure, yet Gordon Ramsay of the Media Standards Trust argues that in replication the report’s findings don’t stand up.
A fortnight ago, the Centre for Policy Studies (CPS) published the report ‘Bias at the Beeb: A quantitative study of slant in BBC online reporting’, announcing, in the words of its author, that “Our results suggest the BBC exhibits a left-of-centre bias in both the amount of coverage it gives to different opinions and the way in which these voices are represented.” Confirmed proof of bias in BBC news – hinted at anecdotally and suggested in individual policy areas in the past – would and should be a big deal, so unsurprisingly the report received substantial coverage in the press. So far, 12 articles have been published citing the conclusions of the report, including a leader in the Daily Telegraph. However, even a cursory look at the research shows fundamental flaws that raise questions about the conduct and presentation of think-tank research and seriously challenge the report’s conclusions.
First, a caveat – this is not intended to be a defence of the BBC; it is an argument for adhering to the basic principles that keep empirical research ‘honest’ – verifiability, replicability, validity of methods, and so on. The CPS report on the BBC is deficient in all of these areas. It is opaque, it omits key data, resists replication, and presents results obtained via contentious methodology as fact without acknowledging alternative explanations, yet it sits on the record as definitive evidence of BBC ideological bias.
The CPS report makes two substantial conclusions. The first is that a statistical analysis using ordinary least squares (OLS) regression and correlation shows that the BBC demonstrates a significant bias towards the left in its coverage of think tanks. This part of the report has already received criticism in this blog by FT journalist Chris Cook and elsewhere: the Guardian and Telegraph are used as benchmarks for ‘left’ and ‘right’, a contestable starting-point that then is used to generate a left-right spectrum of 40 think tanks. Notwithstanding the conceptual problems of a linear model of ideology (particularly with religion-oriented think tanks Ekklesia and Theos in the mix), the ranking contains some spectacular anomalies. For example, Demos sits at the leftmost extreme, with the IEA and Adam Smith Institute far to the left of the IPPR, itself only a few places from the furthest right. The CPS’ author defends this as “not ideal, [but] the ranking above is remarkably consistent with conventional wisdom considering it required no value judgements other than the uncontroversial assumption that The Daily Telegraph is more right-of-centre than The Guardian.”
This, to put it mildly, is not good enough. A research methodology that generates such a flawed model (from which all subsequent results in that section follow) should not be utilised because it fits roughly with the decidedly un-empirical concept of ‘conventional wisdom.’ It should be rejected outright and replaced. Regardless of how well constructed a statistical analysis is (and there is no call to doubt the author’s credentials in this area), flawed variables produce flawed outcomes.
The second part of the CPS report is more intriguing: the CPS eschews the 40-think tank spectrum and takes a smaller subset of ten think tanks, with five from the right, four from the left and one (CentreForum) from the centre. They then analyse mentions of those think tanks in BBC Online news articles between 1st June 2010 and 31st May 2013 (the period of analysis for the whole report) and code them for descriptions as “independent” and whether they were contextualised with ‘health warnings’, which they allocate to three measurable categories:
- An existing position on the issue in question (e.g. “arguably the most vocal think-tank opposed to”, “which favours a free market approach to”);
- An ideological position (e.g. free market, left-of-centre, right-of-centre); or
- An affiliation with a member of a political party.
The results are fairly striking: think tanks on the right are more likely to have a ‘health warning’ attached, and are almost never described as “independent” (NB: each of the ten think tanks describe themselves as “independent” on their websites). However, despite leading the report with a complex statistical analysis, the CPS leaves out any absolute numerical data here, and relies solely on proportions, which should set alarm bells ringing for anyone analysing empirical research.
In order to dig deeper into the results obtained by the CPS, I have tried to replicate this part of the research. Of course, without access to the CPS dataset there will be some differences from their results, but the methodology I have applied is designed to be transparent and robust in its own right.
The Missing Piece in the BBC Online Coverage Data
All BBC Online text-based news stories between 01/06/2010 and 31/05/2013 were analysed; detailed sampling and selection methods listed here. This generated a dataset of 364 stories, which can be accessed, analysed or corrected here. Despite some issues with implementing the CPS methodology which may slightly lower the number of ‘health warnings’ recorded (their first ‘health warning’ is not always mutually exclusive from the second, and it was difficult to distinguish statements about previous research of think tanks from statements about their existing positions, plus confusion over where existing positions on a given policy counts as negative contextualisation), the ‘health warning’ proportions I found are relatively similar to those of the CPS (see Table 1; full results available online here.):
Table 1: Think Tank Stories on BBC, 01/06/2010 – 31/05/2013
This backs up the CPS conclusion that health warnings are more likely to be applied to right-wing think tanks (shaded blue), which are also less likely to be described as “independent”. However, if one includes the total number of stories on think tanks (numbers that it is inconceivable the CPS didn’t have to hand), a completely different picture emerges. Of 364 mentions of think tanks, 55% (199) were of one of the five right-wing think tanks, while 45% (165) were of those of the left-wing or centre. Further, almost equal proportions of each group’s mentions (34% ‘Right’ vs 35% ‘Left/Centre’) were concerning explicit policy proposals or research reports by that think tank. While the contextualisation data (‘health warnings’ and descriptions of independence) certainly mirror the CPS findings, their decision to exclude their numbers of observations thoroughly undermines their overall conclusion that coverage of think tanks betrays a left-wing bias.
What the Left-Right ‘Benchmarks’ Actually Show
So what about their characterisation of The Guardian and The Daily Telegraph as the benchmarks of left and right? Figure 1 ranks the left-to-right balance of all national newspapers, based on a similar calculation of mentions of the ten think tanks over the same time period, using Factiva to gather news stories in preference to LexisNexis as used by the CPS (sampling methods available here); full results for each newspaper can be accessed here. Of course, this is not the same as the statistical analysis of the CPS report; rather, it is a fairly simple calculation of how frequently each newspaper (and BBC Online) mentioned each of the major think tanks used in the second half of the CPS report.
Figure 1: Left-Right Balance, National press and BBC Online, based on mentions of the 5 left/centre and 5 right think tanks
The results indicate that the BBC, with a 45%-55% left-to-right balance lies more or less in the middle, between the Guardian, and the total of all newspapers. These figures suggest that it is actually harder for left-wing think tanks to get a hearing in the bulk of the UK media than it is for right-wing think tanks.
These numbers are all indicative, prior to any reliability testing, and, of course, they do not realistically provide evidence of the existence or absence of bias in the BBC any more than the CPS figures do. What they do represent is a fairly simple way to debunk the main conclusions drawn by the CPS from their own analysis and then spread by an overly-credulous press.
In an age of abundant digital capacity and easy online publication, there is no real justification for not placing raw data in the public domain for open scrutiny, except in extreme cases or sensitive policy areas. This should be normal practice for think tanks, organisations which are specifically designed to add to the stock of public knowledge in circulation. Failure to do so, given the lack of scrutiny on a par with the peer-review process that any comparable piece of academic research would have to pass, will only harm the credibility of think tanks in a big-data era.
In the case of the BBC, we are approaching a period in which public and political knowledge will be instrumental to the various processes of Charter renewal, the government’s plurality review, and changes to communications legislation. It is all the more important, then, that information and research on the Corporation is reliable and freely available.
This originally appeared on the LSE’s Media Project Blog and is reposted with permission.
Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.
Gordon Ramsay is Research Fellow at the Media Standards Trust
