SAGE uses a set of assumptions called the ‘reasonable worst-case scenario’ in its pandemic planning. Jonathan Birch (LSE) looks at the group’s minutes and documents from early 2020 and argues that over-reliance on these assumptions led to costly delays.
What unfolded in the UK in the spring of 2020 was a national tragedy within the global tragedy of COVID-19. The course the epidemic took was not inevitable: it was affected by policy choices made in a state of severe uncertainty. Can we learn from the way those decisions were made to develop better norms for scientific advising in extremis?
The advisory process in the UK has been impressively transparent. In particular, SAGE (the Scientific Advisory Group for Emergencies) has published minutes and other documents soon after its meetings. This gives us a rich set of resources on which to draw. In a draft paper, I discuss several issues arising from these minutes. Here, I want to zoom in on one specific issue: the role played by “reasonable worst-case scenarios” in strategic planning.
The reasonable worst-case scenario
The “reasonable worst-case scenario” (RWCS) has been at the core of SAGE’s approach to the pandemic from the beginning. An RWCS is a set of assumptions that reflect one way in which the epidemic in the UK may unfold. The set of assumptions is “reasonable” in the sense of being regarded by SAGE as a serious possibility. It is “worst-case” in the sense of being at the pessimistic end of the range of serious possibilities.
The implicit principle behind the use of RWCSs seems to be this: if you assume you are in the RWCS, and plan accordingly, then you will be as well prepared as possible for less severe scenarios.
What was the RWCS that guided the UK’s initial response to COVID-19? It was, in some respects, truly grim. SAGE’s RWCS planning assumptions, dated 6 March, set out a scenario in which 80% of the population gets COVID-19 over a period of about nine weeks, with 50% displaying symptoms, and an infection fatality rate (IFR) of 1%. The result would have been around 520,000 excess deaths within three months. In an earlier draft dated 4 March, there is an explicit assumption that no effective treatments or vaccines will become available either before or during the epidemic. This line is deleted from the version dated 6 March, but seems to tacitly guide strategic planning.
SAGE also made some background assumptions that, while not formally part of the RWCS, are important for understanding it. First, SAGE assumed that contact tracing would cease to be effective once there were more than 50 cases per week. Second, SAGE appears to have assumed that, as a realistic maximum, interventions (such as social distancing and shielding) could be sustained for 13 weeks. Some modelling was done of a scenario in which they are sustained for 26 weeks, but 13 weeks is the assumption in the summary papers produced by SAGE. Third, SAGE assumed partial compliance. For example, it was assumed that 50% will comply with household quarantine. This is described in a 3 March paper as “high levels of compliance”.
Yet in other respects, the RWCS assumptions are excessively optimistic. SAGE (in the 4 March draft) assumes that R0, the basic reproductive rate of the virus in the absence of mitigation, is 2.4, leading to a doubling time of 4-6 days. Estimates varied a great deal at the time, and still do, but this was, even then, towards the lower end of serious estimates for R0.
A study published in The Lancet on 31 January had estimated R0 at 2.7. On 11 February, researchers at the Theoretical Biology and Biophysics unit at the Los Alamos National Laboratory, USA, released a preprint estimating that R0 was between 4.7 and 6.6.
There was another optimistic background assumption, introduced on 25 February, that surveillance “should provide evidence of an epidemic around 9-11 weeks before its peak”. Just as R0 being higher than expected was not part of the RWCS, surveillance being poorer than expected was also not part of the RWCS. In short, the planning assumptions at this time were a mix of bleak pessimism and excessive optimism.
Consequences
What were the consequences of these choices? The costly delays between 2 March and 23 March, when cumulative infections in England surged from about 13,500 to about 1.6m (to use the latest MRC-BSU estimates), can be seen as consequences of planning assumptions made around the beginning of this period. The combination of an optimistic estimate for R0, optimism about surveillance, and an assumption that 13 weeks was a realistic maximum duration for any intervention, made it seem optimal to delay the introduction of social-behavioural interventions that were in fact urgently needed.
This problem was compounded by continued scepticism, up to 16 March, about the idea of maximally aggressive suppression. For a time, SAGE’s advice was explicitly against maximally aggressive suppression and in favour of more moderate measures. There was a lot of talk in the media, in early March, about “herd immunity”. SAGE was perceived by its critics to be following a “herd immunity strategy”. The Chief Scientific Adviser (Sir Patrick Vallance) vehemently denied that there was any such strategy in private emails subsequently released to the BBC.
What was really going on? The reality is clear from the documents released by SAGE. Up to 9 March, there was no explicitly recommended strategy. SAGE presented various different options, without formally endorsing one. The recommended strategy between 9 and 16 March was a “high transmission reduction strategy” in the sense of Figure 1: a strategy which aimed to flatten the curve without suppressing it completely. In the week of 16 March, the recommendation changed abruptly to one of maximally aggressive suppression, including school closures.
Figure 1: A sketch that features prominently in a paper dated 9 March, intended to highlight the superiority of “high transmission reduction” to “very high transmission reduction, later lifted”.
This initially sceptical attitude towards maximally aggressive suppression can be traced back to the RWCS planning assumptions. In a reasonable worst-case, no effective treatment, contact tracing system or vaccine becomes available before measures are relaxed, and no long-term behavioural changes are instilled, so total suppression leads robustly to the epidemic returning with unmitigated force in the autumn, infecting 80% of the population and overwhelming the health service. This corresponds to the green line in Figure 1. In that bleak scenario, we come to bitterly regret the aggressive measures adopted in the spring.
In the actual world, by contrast, we have been left bitterly regretting our failure to take these measures soon enough. If SAGE had unambiguously recommended maximally aggressive suppression (including school closures) on 2 March, when its SPI-M subcommittee first reported that “It is highly likely that there is sustained transmission of COVID-19 in the UK at present”, then many lives would have been saved and the epidemic would have taken a different course.
Reflections
Let’s return to the implicit principle guiding the use of RWCSs: if you assume you are in the RWCS, and plan accordingly, then you will be as well prepared as possible for less severe scenarios. This case, I suggest, shows us some important exceptions to this principle.
The principle is not true if your RWCS is pessimistic in most relevant respects but not all of them. I will introduce the term “globally pessimistic” for a scenario that is pessimistic in all relevant respects. If the RWCS is not globally pessimistic (e.g. its estimate for R0 is too optimistic), then there is a serious risk that reality will be worse than the RWCS in some respects. This is what happened in relation to R, which was almost certainly above 2.4 before the spring lockdown in England, according to more recent MRC-BSU modelling, despite substantial attempts at mitigation.
Of course, a globally pessimistic scenario may well seem unreasonable. It is tempting to think: it is very unlikely that we will be unlucky in all respects! This brings out a tension in the concept of an RWCS. To avoid reality catching you out, the RWCS must be globally pessimistic, and may therefore look unreasonable when viewed as a whole. But what matters is that each individual assumption represents a serious possibility.
More subtly, there can be circumstances in which assuming you are in the RWCS justifies actions, delays, or omissions that will impair your response significantly if you are in a less severe scenario.
For example, it makes sense to say that, in a reasonable worst case, no effective treatment or vaccine will become available, contact tracing will never become effective, and no long-term behavioural changes will be instilled, even if you delay the epidemic by several months. In this case, maximally aggressive suppression of transmission is likely to make things worse in the long run, as depicted in Figure 1.
But suppose you are in a less severe scenario, in which one or more of these pessimistic assumptions is false. In such a scenario, maximally aggressive suppression is likely to be far superior to more moderate action, in terms of both its public health consequences and its long-term economic consequences. If you plan for the RWCS, and are thereby led to adopt a strategy that involves aiming for the brown/orange line in Figure 1, you will not be as well prepared as possible for a less severe scenario.
Proposals
This leads me to some proposals concerning the use of RWCSs in scientific advice. First: RWCSs, if used in any form of planning, should be globally pessimistic, which is to say at the pessimistic end of scientific opinion in all potentially relevant respects, not just some.
Second, although they are a useful guide for day-to-day operational planning (e.g. how many pieces of PPE to purchase), RWCSs should not dominate strategic planning. For strategic planning, it is important to consider a wide range of possible scenarios, including but not limited to the reasonable worst case (which, if globally pessimistic, will be unlikely).
The apparent inevitability of a large wave as soon as measures were relaxed was sensitive to a specific set of worst-case planning assumptions, which assumed that measures could not be sustained until an effective treatment, vaccine, or contact tracing system was implemented. These assumptions dominated strategic planning up to 16 March.
Yet as soon as modellers dared to relax one of those pessimistic assumptions, the strategic picture suddenly changed. A paper by modellers at Imperial College, dated 16 March, considered the possibility of maintaining maximally aggressive suppression measures (including long periods of school closure) indefinitely, until a vaccine is achieved (Figure 2). The paper concludes, dramatically, that aggressive suppression is “the only viable strategy at the current time”. This paper produced a swift change in the direction of the UK’s response.
Figure 2: A graph from Ferguson et al. (2020), in which aggressive suppression measures are maintained indefinitely. The blue-outlined blocks represent periods of school closure.
This leads me to another proposal: scientific advisers should highlight, as part of their advice, the circumstances under which their recommended actions might lead to serious regret. When SAGE recommended a “high transmission reduction” strategy over maximally aggressive suppression on 9 March, it should have highlighted the potential for this strategy to lead to serious regret—in the form of tens of thousands of excess deaths which could, in retrospect, have been prevented by pursuing a more aggressive suppression strategy at an earlier stage.
These three proposals (make the RWCS globally pessimistic, do not allow it to dominate strategic planning, and communicate the circumstances under which recommendations arising from it may lead to serious regret) are not a recipe for effective scientific advising. A lot can still go wrong. But I do think they would have helped in this case – and might help in future crises on a similar scale.
This post represents the views of the author and not those of the COVID-19 blog, nor LSE.
4 Comments