LSE - Small Logo
LSE - Small Logo

Blog Admin

April 24th, 2013

Reinhart-Rogoff revisited: Coding errors happen – key problem was in not making the data openly available from the start

16 comments | 3 shares

Estimated reading time: 5 minutes

Blog Admin

April 24th, 2013

Reinhart-Rogoff revisited: Coding errors happen – key problem was in not making the data openly available from the start

16 comments | 3 shares

Estimated reading time: 5 minutes

velichka copyThe eventual replication of the data from the Reinhart-Rogoff paper on 90% debt/GDP threshold has sparked vibrant discussion on the impact of error-ridden research on austerity policies around the world. Velichka Dimitrova argues this controversy highlights the importance of open data of economics datasets. Coding errors happen, yet the greater research problem was not allowing for other researchers to review and replicate the results through making the data openly available as early as possible.

Another economics scandal made the news last week. Harvard Kennedy School professor Carmen Reinhart and Harvard University professor Kenneth Rogoff argued in their 2010 NBER paper that economic growth slows down when the debt/GDP ratio exceeds the threshold of 90 percent of GDP. These results were also published in one of the most prestigious economics journals – the American Economic Review (AER) – and had a powerful resonance in a period of serious economic and public policy turmoil when governments around the world slashed spending in order to decrease the public deficit and stimulate economic growth.

Carmen Reinhart

Kenneth Rogoff

Yet, they were proven wrong. Thomas Herndon, Michael Ash and Robert Pollin from the University of Massachusetts (UMass) tried to replicate the results of Reinhart and Rogoff and criticised them on the basis of three reasons:

  • Coding errors: due to a spreadsheet error five countries were excluded completely from the sample resulting in significant error of the average real GDP growth and the debt/GDP ratio in several categories
  • Selective exclusion of available data and data gaps: Reinhart and Rogoff exclude Australia (1946-1950), New Zealand (1946-1949) and Canada (1946-1950). This exclusion is alone responsible for a significant reduction of the estimated real GDP growth in the highest public debt/GDP category
  • Unconventional weighting of summary statistics: the authors do not discuss their decision to weight equally by country rather than by country-year, which could be arbitrary and ignores the issue of serial correlation.

The implications of these results are that countries with high levels of public debt experience only “modestly diminished” average GDP growth rates and as the UMass authors show there is a wide range of GDP growth performances at every level of public debt among the twenty advanced economies in the survey of Reinhart and Rogoff. Even if the negative trend is still visible in the results of the UMass researchers, the data fits the trend very poorly: “low debt and poor growth, and high debt and strong growth, are both reasonably common outcomes.”

Source: Herndon, T., Ash, M. & Pollin, R., “Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff, Public Economy Research Institute at University of Massachusetts: Amherst Working Paper Series. April 2013.

What makes it even more compelling news is that it is all a tale from the state of Massachusetts: distinguished Harvard professors (#1 university in the US) challenged by empiricists from the less known UMAss (#97 university in the US). Then despite the excellent AER data availability policy – which acts as a role model for other journals in economics – AER failed to enforce it and make the data and code of Reinhart and Rogoff available to other researchers.

Coding errors happen, yet the greater research problem was not allowing for other researchers to review and replicate the results through making the data openly available. If the data and code were available upon publication already in 2010, it may not have taken three years to prove these results wrong – results which may have influenced the direction of public policy around the world towards stricter austerity measures. Sharing research data means a possibility to replicate and discuss, enabling the scrutiny of research findings as well as improvement and validation of research methods through more scientific enquiry and debate.

The Open Economics Working Group of the Open Knowledge Foundation advocates the release of datasets and code along with published academic articles and provides practical assistance to researchers who would like to do so. Get in touch if you would like to learn more by writing us at economics [at] okfn.org and signing for our mailing list.

Link to paper

Link to data and code

Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of the London School of Economics.  

About the Author

Velichka Dimitrova is project coordinator of Open Economics at the Open Knowledge Foundation. She is based in London, a graduate of economics (Humboldt Universität zu Berlin) and environmental policy (University of Cambridge) and a fellow of the Heinrich Böll Foundation. She can be found on Twitter at @vdimitrova

Print Friendly, PDF & Email

About the author

Blog Admin

Posted In: Academic publishing | Impact

16 Comments