Or would a Research Methods Annex be better?

Image credit: CC0 Public Domain.

It is axiomatic that any doctorate (and any serious academic book) must be based on sound methods, systematically applied. PhD examiners especially (but also critical readers of academic work more generally) must never be left in any doubt about exactly how findings and results have been generated, their precise provenance and meaning. That much is common ground.

But how should this requirement best be implemented? The near-universal answer across academia at present is that any ‘big book’ PhD thesis must include a separate Methodology chapter, usually positioned immediately after the theory and literature review — normally either at Chapter 2 (where both these other functions are combined in the first chapter) or at Chapter 3 (where the other two functions get a chapter each).

The costs of a ‘Methodology’ chapter

What can go wrong with the standard approach?

  • Advance chapters explaining methods are often some of the dullest pieces of text ever written. In routine cases the chapter says little more than: ‘I used this perfectly well known method and implemented it in the pretty well standard way, barring a couple of difficulties on the way’. Yet, because it is a whole chapter, authors often feel they have to inflate this message to 6,000+ words and ‘engage with the literature’ to explain why they used a perfectly legitimate and uncontroversial approach. Cue long discussions of method ‘isms’ and perhaps epistemological/philosophical macro-perspectives that are really light years removed from the actual problem in hand. If you are good at this kind of thing, the damage may not be too great: you’re ‘waving not drowning’. But if you are really an empiricist ‘coming up for air’ at this level of discussion, then it is all too easy to get you out of your depth, creating potential oral exam (viva) time-bombs with more neurotic thesis examiners. And even if nothing here is a liability, still in terms of the ‘Build, Blur, Corrode’ criteria, a whole chapter of ‘blur’ is a dead weight you could do without.
  • The other great theme of many a methodology chapter is the pre-emptive defensive move: ‘I did the best I feasibly could, given the limitations of the available data’. This may well be perfectly true. But at this stage readers have not seen even a single page of substantive analysis showing what you actually accomplished. So reading in isolation about all the smallish potential problems you faced makes for depressing reading. At the start of a thesis or book your status and competence as a researcher also remains to be established for readers, because they just don’t know if the methods choices you made worked out well, or not.
  • Focusing on Methodology in a whole-chapter format also creates a common dilemma for authors. On one fork they stay rather high-level and vague. They keep the text more consistent with the high theory impetus of the opening chapter(s) and the launch into substantive analysis that comes next, by writing only a short chapter (6,000 words or less), devoid of quite a lot of needed detail. On the other fork of the dilemma authors cram in lots of specific information alongside the major-method themes, so that it’s all in one place — but often not in an understandable form. For instance, you may tell us how you recoded variables x and y in an abstract way, before we know how the data on these variables are patterned. The progression of ideas and arguments in your thesis or book stalls in the mud of a hundred detailed battles we have not even seen fought yet.
  • All these problems often create a low-energy (and often not very relevant) zone in your thesis, dashing the readers’ high hopes built up in your carefully-crafted and motivating first chapter. For experienced (=jaded) PhD examiners a ‘standard’ method Chapter simply prolongs the pages of text that they have to wade through before they see one original thing they did not know already. Count the pages involved in the theory introduction, literature review and methods chapters. If it’s more than 50 the examiners are yawning. If it’s getting towards 100 pages they’re pretty impatient.
  • And if you send your PhD to a publisher as a possible monograph, the inclusion of a low energy Methods chapter at 2 or 3 on the Contents page shouts ‘dull PhD’ at them. They dig out the well-used reject letter that explains that current profit margins being so low in academic publishing (i.e only 35%), they really cannot take on the risk of publishing your eminently serious work, much to their regret.

But, you say, it’s my duty to inform colleagues of exactly what I did in my study. This is true, but does it have to be at chapter length, in chapter style and located in the main run of chapters at this early stage of the argument? The key tests here are these questions: Are your methods innovative? Do they solve a problem that previously stymied science or scholarship? Are methodology changes or variations or innovations part of the core value-added of your PhD or scholarly book? If you can put your hand on your heart and say Yes to any of these, then your thesis needs a methods chapter. Stick to the standard approach, because for you methods innovation and originality is so central and integral to the work done. But if not, there is an alternative, with many practical advantages for most authors.

The alternative: focused methods information and a Research Methods Annex

Doing without a Methodology chapter does not mean leaving your methods obscure, but presenting this essential information in a different way. Again it bears repeating that PhD examiners and readers of any serious academic work must feel fully informed about what you did and why. In this approach that means they learn what you did at three key points:

  1. In your opening, theory and problem-framing chapter you need to cover any major methods issues briefly in the last section of your first chapter. This is natural progression since in academia methodological approaches are often integral to major theory controversies or differences. Here you make clear where you stand and what broad approach you have followed.
  2. More specific methods information (but still focusing on important issues) is given in subsequent chapters, very close to the presentation of substantive findings. Readers and examiners cannot normally assess the viability of a methodological approach until they see what results it produces — the findings themselves are the essential cue we need to see what might have gone right or wrong. Equally it’s much easier for you to make a plausible case that your methods choices were correct by discussing substantive evidence, showing that you have undertaken sensitivity analyses or that different ways of tackling problems produce convergent solutions. These things you can only assert in an early-on Methodology chapter, but here you can demonstrate them. Note too that this approach is what you have to do in publishing thesis chapters as journal articles (‘paperizing’ them). And if you are not doing a ‘big book’ PhD but a ‘papers model’ doctorate then you will have to do this anyway.
  3. At the end of the main chapter sequence, at Annex 1, is a comprehensive and carefully written Research Methods Annex. This does not have to be written in chapter style, nor is it constrained in terms of length or the level of detail included. Instead it aims to give an extremely detailed information to any other scholar or scientist following in your footsteps, going as long as needed to provide exact guidance on the methods you used and the various decisions on how to proceed that taken at each stage of analysis. A well done Annex will normally be far more detailed and directly useful than a discursive methods chapter placed early in the main sequence. And because it is located at the end of the analysis, when readers have already seen your findings in detail, you save time and have additional authority as a researcher to draw on.

Producing a detailed, step-by-step Research Methods Annex fits well with the modern recognition of the powerful role played by ‘tacit knowledge’ in scientific and scholarly work. It invites you to try and set out everything that a reader needs to know in order to replicate your approach. You will not succeed in making everything explicit, according to philosophers of science. But you will get a lot further down the road than just writing an introductory Methodology chapter. Undertaking this Annex can take some time that you may begrudge in the last, rushed months of your doctorate or book. But doing so will also meet the increasing demands from journals that you provide a replicable data set to back up your paper.

A Research Method Annex also chimes with the open science drive to publish datasets as free-standing entities, attracting scholarly credit in their right. Of course, if datasets are to be reused, and to voyage alone in the world, then how variables are coded and indices are constructed and re-codings are accomplished needs to be crystal clear.

Heresy, supervisors and examiners

Yet, you are probably thinking, academics are conservative folk. This may all sound very well in theory, but in my discipline and university people are ultra-conventional. My supervisor, my department and any conceivable set of examiners all make completely clear that Ch.2 or 3 must cover Methodology. Oftentimes that assumption is built into the reporting forms that departments use. Older supervisors did it this way when they were doing PhDs, and perhaps they’re resistant to changes. Not having a Methodology chapter to them may seem like a heresy, for which I’d still get burnt at the stake.

Well, this is a factor and I don’t want to deny it’s there, especially in disciplines that adhere a lot to conventional macro-structures for the chapter sequence in PhD theses in more quantitative social sciences and many STEM (science, technology, engineering and mathematics) disciplines. My advice here is to discuss the ideas in this post with your supervisors and colleagues, and see what they think in relation to your topic.

Meanwhile, across most of the humanities and the ‘soft’ social sciences I think resistance to doing creative writing better has now weakened a lot. Most academics now accept that PhD theses and books should be written with subject-specific structures optimizing argumentative development, and written well for maximum academic impact. And when it comes to publishing your successful thesis as book — yes, this is an eminently realizable goal in these disciplines — you don’t have to do anything like as much work. Your Research Methods Annex is deposited in your university’s e-publications depository, and URL referenced in the book at all key points, allowing the text to be focus consistently on the main arguments and to strike and maintain a constant developmental tempo throughout.

So, if you think this approach can work for your thesis or book, good luck in breaking the (old, stale) mold.

To put these ideas in a wider context, you might find it helpful to read parts of my book: Patrick Dunleavy, ‘Authoring a PhD’ (Palgrave, 2003), especially the chapter on the macro-structure of a thesis. See also useful material on the LSE’s Impact blog and on Twitter: @Write4Research.