A recent report by LSE academics extolling the benefits of competition between NHS hospitals claims causality where there is none. Allyson Pollock, Alison Macfarlane and Ian Greener argue that the authors engage in data dredging and faulty empirical analysis. In so doing, they sweep aside decades of evidence showing why markets do not work in health services and lend support to an HSC Bill that is inherently dangerous.

The drip feed of pro-competition studies from Zack Cooper at LSE raises serious questions for the academic community and the public about what constitutes bad science and what to do about its politicisation. Recently, on 21 February in the columns of the FT, Cooper and colleague Julian Le Grand warded off serious scientific criticisms of the studies with an ad hominem attack, categorising those in favour of competition as empiricists and those whose work is critical of markets in health care as intuitivists. In so doing they sweep aside decades of careful economic theory and evidence which shows why markets do not work in health services and distract the reader from the facts that their work is ungrounded and far from empirical. Their repeated claims that competition in the NHS saves lives and improves quality and productivity have no scientific basis.

In July 2011, Cooper and colleagues at the LSE press-released an unpublished paper to coincide with the prime minister’s announcement on the Future Forum which had been set up in response to deep public concerns about the Health and Social Care (HSC) Bill. These concerns resulted in the government suspending the legislative process for two months to under take a ‘listening exercise’ with the public. The FT and The Economist put their paper centre stage in the HSC Bill debate. The authors were sufficiently persuasive for the prime minister to declare that “competition is one way we can make things work better for patients. This isn’t ideological theory. A study published by the London School of Economics found hospitals in areas with more choice had lower death rates.” The study in question claimed that “using [acute myocardial infarction] AMI mortality as a quality indicator, … mortality fell more quickly (i.e. quality improved) for patients living in more competitive markets after the introduction of hospital competition (to the NHS) in January 2006”.

The major improvements in outcome after acute myocardial infarction can be attributed to improvements in primary prevention in general practice and in hospital care, including the introduction of percutaneous IV angiography. The government’s own cardiac Tzar, Sir Roger Boyle, was sufficiently angered by their claims to respond with withering criticism: “AMI is a medical emergency: patients can’t choose where to have their heart attack or where to be treated!” It is “bizarre to choose a condition where choice by consumer can have virtually no effect”. Patients suffering “severe pain in emergencies clouded by strong analgesia don’t make choices. It’s the ambulance driver who follows the protocol and drives to the nearest heart attack centre”.

The intervention that the authors claimed reduced heart attacks and was a proxy for competition was patient choice. In 2006, patients were given choices of hospitals including private for-profit providers for some selected treatments. Less than the half patients surveyed in 2008 even remember being given a choice, and only a tiny proportion made those choices based on data from the NHS choices website. If patient choice was one of the two key elements of competition, it wasn’t prevalent and rather than being derived from the authors’ data, it was assumed.

Crucially, even if patient choice had occurred it does not explain why heart attack mortality rates fell. There is no biological mechanism to explain why having a choice of providers for elective hip and knee operations surgery (including hospitals which did not treat or admit acute MI patients) could affect the overall outcomes from acute myocardial infarction where patients do not exercise choice over where they are treated.

The problem of data dredging is well known; if you repeat an analysis often enough significant statistical associations will appear. But the authors make the cardinal error of not understanding their data and of confusing minor statistical associations with causation. Deaths from acute MI are not a measure of quality of hospital care, rather a measure of access to and quality of cardiology care. At best, what the paper appears to show is not the effect of choice on heart attacks but that if an individual has a heart attack in an area close to a hospital and their GP is near the hospital, then outcomes are better, but such findings are not new.

Cooper’s working paper which the government cited as supporting their reforms was subsequently published in the Economic Journal. That it got through that journal’s peer-review process is perhaps indicative of the poor understanding of healthcare and routine data from reviewers of that journal. Our response to their flawed work was published in a peer-reviewed piece in the Lancet. They responded with mainly ad hominem attack and we again responded with scientific criticism.

Last week Cooper and colleagues were at it again with another working paper (as yet unpublished in an academic journal) that was once again miraculously timed to coincide with an important event; the prime minister’s summit on the NHS Bill. This time the authors claim that length of stay fell more rapidly in NHS hospitals experiencing greater competition and that the risk of cherry picking by the private sector made a case for risk adjusted price. Once again, the authors were careless with the data and the study design.

There are three problems with their analysis of the data: they seem unaware that lengths of stay differ between the conditions they examine; they ignore the political context in which the data was generated; and finally, they show little knowledge of the particularities of the conditions they include and how these will affect the data.

Cooper and his colleagues use the average length of stay for four conditions, elective hip replacements, knee replacements, hernia repairs and arthroscopies, each of which differs widely in lengths of stay. Arthroscopy is usually done as an outpatient and may not be recorded on hospital episode statistics. Hernia repair is usually a day case although the average overall length of stay varies by type of procedure and with median lengths of stay of one or two days. In contrast, hip and knee replacements have median lengths of post-operative stay of four or five days again depending on the procedures and morbidity, with average lengths of stay in 2010-11 ranging from 5.9 to 8.2 days for hip replacements and 5.5 to 5.8 for knees. (See the Information Centre inpatient and outpatient data)

Thus, if providers have switched to arthroscopies and hernia repairs or to operating on patients who are well and healthy they will appear to have shortened their pre-operative and post-operative length of stay to less than a day. So a provider’s length of stay will depend on the mixture of operations and mixture of patients and how far they travel. The authors appear to have made no attempt to examine differences in case mix and length of stay. This is a serious error.

Equally, the authors do not look at how clinical coding changed following the introduction of the tariff in 2006. Gaming, upcoding and diagnostic drift are widely recognised in research on the NHS in the 2000s, with providers seeking to improve and increase their payments through fraudulent billing and accounting. This will apply especially where hospitals are under severe financial pressures and have strong motivations and perverse incentives to change the coding procedures. Arthroscopy procedures, which previously have been coded as an outpatient activity or not at all (i.e. it would not have been counted as an admission), may now be recorded separately as a day case inpatient procedure. These changes in coding distort measures of productivity so that providers may appear to be more efficient as they appear to do more work than they actually do.

Finally, length of stay is also a product of a range of factors related to the conditions in their data; pre-operative work for hip and knee replacement needs to take account of rurality and patient fitness for discharge, especially if patients live alone and have other co-morbidities and complexities. Patients who live close to a hospital may come in as an outpatient, while patients who live some distance away may require overnight stays. The authors should also have looked at readmission rates; premature discharge can result in readmission. The authors have not attempted to examine any of these factors, and neither have they considered the effect of hospital concentration on their data.

Le Grand and Cooper call themselves ’empiricists’ and all those that disagree with them ‘intuitivists’. Unlike scientists, however, they have made no ‘real life’ observations themselves from which they have generated their theories. They do not appear to have the basic understanding of clinical practice. They have not made predictions, tested their theories with experiments, or adapted their models to see if they can do anything other than provide one explanation of many that could be derived from historical data. Moreover, they ignore the factors that underpin the generation of data and the need to understand how it is constructed and shaped. Data dredging has resulted in statistical associations but association is not causation. Bad science makes bad policy and bad policy leads to careless talk. Careless talk will cost lives especially when it is used in support of the HSC Bill.

Please read our comments policy before posting.

Replies to this article were subsequently provided separately by Julian Le Grand and by Zack Cooper, Steve Gibbons, Simon Jones and Alistair McGuire. You can read Le Grand here, and Cooper et al here.  

About the authors

Allyson Pollock is Professor of Public Health Research and Policy at Queen Mary, University of London. She set up and directed the Centre for International Public Health Policy at the University of Edinburgh from 2005 to 2011, and prior to that she was Head of the Public Health Policy Unit at UCL and Director of Research & Development at UCL Hospitals NHS Trust.

Alison Macfarlane joined City University London in March 2001 to lead research in the Department of Midwifery after working at the National Perinatal Epidemiology Unit since the late 1970s.  Her current research assesses the impact of new birth centres on choice and outcome of maternity care in inner city areas.

Ian Greener is Deputy Head of the School for Postgraduate Training and Research in the School of Applied Social Sciences. He is also an Assistant Director of the Wolfson Research Institute at Durham.