LSE - Small Logo
LSE - Small Logo

Romesh Vaitilingam

October 25th, 2021

Natural experiments in labour economics and beyond

0 comments | 19 shares

Estimated reading time: 3 minutes

Romesh Vaitilingam

October 25th, 2021

Natural experiments in labour economics and beyond

0 comments | 19 shares

Estimated reading time: 3 minutes

The 2021 Nobel Prize in Economic Sciences was recently awarded to David Card of the University of California, Berkeley, ‘for his empirical contributions to labour economics’, and to Joshua Angrist of MIT and Guido Imbens of Stanford University ‘for their methodological contributions to the analysis of causal relationships’. How do economists view the work of the new laureates? Romesh Vaitilingam sums up the results of an expert panel survey run by The Initiative on Global Markets. 


 

As has become an annual tradition at the Initiative on Global Markets, we invited our panels of economists to comment on the work of the winners of the Nobel Prize in Economic Sciences. We asked the experts whether they agreed or disagreed with three statements, and, if so, how strongly and with what degree of confidence:

1) The introduction of natural experiments to economic analysis of the labour market and related areas has led to a more precise understanding of cause and effect.
2) The ‘credibility revolution’ in empirical economics has improved our understanding of a number of public policy issues, including education, immigration and the minimum wage.
3) In pursuit of credible research designs, researchers often seek good answers instead of good questions.

Background on the first two questions has been provided in much of the coverage of the new Nobels, including a column by Jörn-Steffen Pischke, who has co-authored books and papers with Joshua Angrist. The third was stimulated by their 2010 Journal of Economic Perspectives paper on the credibility revolution, which closes by discussing ‘the claim that the experimentalist paradigm leads researchers to look for good experiments, regardless of whether the questions they address are important.’

Of our 43 US experts, 41 participated in this survey; of our 48 European experts, 39 participated – for a total of 80 expert reactions.

Statement 1. The introduction of natural experiments to economic analysis of the labour market and related areas has led to a more precise understanding of cause and effect.

A big majority of respondents on both panels agree with the first statement. Weighted by each expert’s confidence in their response, 72% of the US panel strongly agree, 24% agree, 0% are uncertain, and 3% disagree (the totals don’t always sum to 100 because of rounding). Among the European panel (again weighted by each expert’s confidence in their response), 66% strongly agree, 31% agree, 3% are uncertain and 0% disagree.

Overall, across both panels, 69% strongly agree, 27% agree, 2% are uncertain, and 2% disagree.
Among the short comments that the experts are able to include in their responses, Larry Samuelson at Yale says: ‘Natural experiments are a welcome complement to other tools.’ Jan Pieter Krahnen at Goethe University Frankfurt comments: ‘Using natural experiments is an intelligent way to build causal inference on the diversity of institutions, shocks and behaviour.’ And Franklin Allen at Imperial remarks: ‘As long as the natural experiment is well designed, these studies have the potential to increase our knowledge.’

Others mention applications beyond labour economics. Christian Leuz at Chicago replies: ‘Agree but not only in labour and micro, but also many other applied fields like finance and accounting.’ Pete Klenow at Stanford provides a reference to a survey of natural experiments in macroeconomics by European panel member Nicola Fuchs-Schundeln and a colleague. And Christopher Udry at Northwestern links to two experimental studies in development economics.

A few panellists express reservations. Anil Kashyap at Chicago notes: ‘Can’t answer all questions with these approaches, but we have definitely learned a lot.’ Patrick Honohan at Trinity College Dublin observes: ‘Although, as with randomised control trials, the evidence often comes from particular contexts that may not be generalisable.’ And Costas Meghir at Yale, who votes uncertain, adds: ‘Use of natural experiments without models relies on behavioural assumptions that are often unstated. May learn little about mechanisms.’

Statement 2. The ‘credibility revolution’ in empirical economics has improved our understanding of a number of public policy issues, including education, immigration and the minimum wage.

On the second statement about the impact of the credibility revolution on policy debates in education, immigration, and the minimum wage, again there is a big majority in agreement on both panels. Weighted by each expert’s confidence in their response, 60% of the US panel strongly agree, 32% agree, 7% are uncertain, and 0% disagree. Among the European panel (again weighted by each expert’s confidence in their response), 48% strongly agree, 52% agree, and 0% are uncertain or disagree.

Overall, across both panels, 54% strongly agree, 42% agree, 4% are uncertain, and 0% disagree.

Among the comments, Nicholas Bloom at Stanford says: ‘The research on the minimum wage has been pathbreaking – I absolutely changed my mind based on the evidence, and it has driven policy.’ Ricardo Reis at LSE adds: ‘Definitely improved. Of course, one would hope that progress had been even larger and more decisive. But no doubts on the sign of progress.’ And Costas Meghir notes: ‘There has been increased emphasis on justifying sources of exogenous variation in both structural and quasi-experimental studies.’

Several panellists refer to where natural experiments sit relative to other methods in research and policy evaluation. Larry Samuelson comments: ‘I view it as credibility evolution, with techniques that have built upon and evolved alongside other methods.’ Antoinette Schoar at MIT suggests: ‘Policy evaluation is complex and we need to draw on as many methods as possible for progress. But the credibility revolution is a key tool.’ Christian Leuz adds: ‘For policy, causal inferences are critical so need credible designs.’

Robert Shimer at Chicago, who votes uncertain, comments: ‘The devil is in the details with identification; and researchers often explore limited outcomes, e.g. only the short run.’ Also voting uncertain is Angus Deaton at Princeton, who says: Some plusses lots of minuses.’

Statement 3. In pursuit of credible research designs, researchers often seek good answers instead of good questions.

The third question – inviting views on whether in pursuit of credible research designs, researchers often seek good answers instead of good questions – generated considerably more differences of opinion. Weighted by each expert’s confidence in their response, 4% of the US panel strongly agree, 63% agree, 23% are uncertain, 7% disagree and 4% strongly disagree. Among the European panel (again weighted by each expert’s confidence in their response), 17% strongly agree, 36% agree, 37% are uncertain, and 10% disagree.

Overall, across both panels, 10% strongly agree, 49% agree, 30% are uncertain, 9% disagree, and 2% strongly disagree.

Among the comments of those who agree, Franklin Allen notes: ‘Unfortunately, there is some truth in this statement. It’s difficult to identify good natural experiments for many questions.’ Larry Samuelson adds: ‘This is the case with much of economics, and the social sciences more generally.’ And Anil Kashyap responds: ‘Running joke in the faculty lounge after some shock: “we will see a diff-in-diff paper on the job market using that variation next year…”’.

Others comment on incentives within economic research. Aaron Edlin at Berkeley states: ‘The profession continues to place a premium on clever identification strategies.’ Kenneth Judd at Stanford argues: ‘Too many economists focus on tractability, and even demand that one should know what the results will be before doing the analysis.’ And Luigi Guiso at the Einaudi Institute for Economics and Finance observes: ‘There is indeed a risk that the method drives the question and the risk is already real.’

Several panellists demur at ‘often’ in the statement. David Autor at MIT says: ‘This sometimes happens, for sure. But “often” is too high a bar for me. So, I disagree’; and Barry Eichengreen at Berkeley states: ‘If the question had been worded “sometimes” rather than “often” I would have agreed.’ David Cutler at Harvard concurs: ‘There are certainly some studies like this. “Often” is the key word here’; as does Daron Acemoglu at MIT: ‘I would say “sometimes” rather than “often”. Credibility revolution is fantastic for economics but we can/should not sacrifice big questions.’

Other panellists go further. Oriana Bandiera at LSE argues: ‘I agree that some researchers do this but you can’t blame the method, it’s like saying knives are bad because people use them to hurt others.’ Similarly, Antoinette Schoar comments: ‘I believe that methods can indeed shape the questions researchers ask. But this is true for all tools, including structural estimation, etc.’

Still others are critical of the way the question is framed. Richard Thaler at Chicago says: ‘Don’t love the wording here. Researchers try to answer some question, often it is not the most interesting one.’ Pol Antras at Harvard comments: ‘I don’t know what’s meant by “good”. I see a trade-off between credibility and generality (or external validity), but questions aren’t “bad”.’ And Abhjit Banerjee at MIT protests: ‘I really don’t know what that means. Should we pursue good questions which can’t be answered? Wittgenstein had it right.’

In similar vein, Richard Schmalensee at MIT argues: ‘Working on good questions that can’t be given good answers is a waste of effort; researchers aren’t wrong to consider quality of answers.’ And Michael Greenstone at Chicago objects: ‘This is a straw man debating style criticism. Is the alternative bad answers to good questions? Count me out for that!’

A final set of comments portray a trade-off. Ricardo Reis remarks: ‘Within set of good answers, they pick the better questions. Researchers trade off the two, unclear that they do so sub-optimally.’ Nicholas Bloom adds: ‘There is a trade-off between the quality of the question and the quality of the answer, but as long as we are on the frontier, all is good.’ And Olivier Blanchard at the Peterson Institute concludes: ‘The initial phase was indeed good answers. But we have moved over time to good questions.’

♣♣♣

Notes:

  • The survey is conducted regularly on different topics by The Initiative on Global Markets, of the University of Chicago Booth School of Business. All comments made by the experts are in the full survey results for the US and EU panels.
  • The post represents the views of its author(s), not the position of LSE Business Review or the London School of Economics.
  • Featured image by Ryoji Iwata on Unsplash  
  • When you leave a comment, you’re agreeing to our Comment Policy.

About the author

Romesh Vaitilingam

Romesh Vaitilingam is an economics writer and communications consultant, and editor-in-chief of the Economics Observatory. He is also a member of the editorial board of VoxEU and manages the IGM Forum surveys of economic experts.

Posted In: Economics and Finance

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.