COVID-19 has accelerated calls for fast, open science to inform policy responses. However, when contradictory or false results become public, the negative consequences of this becomes hard to contain. Nate Breznau discusses the Heinsberg Study into COVID-19, outlining how the lack of appropriate scientific scrutiny led to policy responses that were misinformed and dangerous. Breznau argues that for fast science to be reliable, it needs to be underscored by access and transparency. Otherwise, it risks becoming fake news.

COVID-19 and the need for fast science

The current Coronavirus Pandemic is an example of human lives depending on fast and open science. Appropriate policy response to the deadly virus demands speed, accessibility and reliability. The speed part is in no shortage, with working papers flooding the preprint servers since January. Unfortunately, a large portion of these results are nothing shy of fake news. Experts are therefore crucial to filtering out the garbage and providing context to allow appropriate interpretation of results. But, experts cannot judge the findings if the results are not transparent. Moreover, it may be too late for experts to filter results once the media starts running away with them.

When unreliable or contradictory results appear in the media, the public are unlikely to adopt new social practices. When exposed to a high degree of uncertainty, the public are unable to make decisions and tend to default to their standard operating behaviours, but doing so could be fatal in a pandemic. The events surrounding the Heinsberg Study in Germany offer us a glimpse of both the problems of fast science, and the solutions of open science. But open science has its limits.

The Heinsberg Study

In North Rhine Westphalia (NRW) the February Carnival season was a ‘super-spreader’ event for SARS-CoV-2 in Germany. A team of virologists led by Hendrik Streek obtained funding from the NRW government to conduct a quick study of those affected from March 30th to April 6th. In remarkable turnaround time they presented results in a press conference on April 9^th, concluding that infection-fatality rate of the virus is 0.37%, far lower than the 2% estimated by Johns Hopkins. They concluded that nearly one-fifth of the public already had antibodies. The key takeway points were that Germany was underway to herd immunity and lockdown measures could be reduced.

The presentation of the Heinsberg Study was done without context. Only one scientist was present who was not part of the Study. There was no preprint working paper or data for public scrutiny and peer review. Thus, the normal filter that regulates dissemination of scientific research was lacking. In emergencies, traditional peer review takes too long. But immediate peer review is possible when other scientists are allowed, if not encouraged, to see the data and results.

This press conference and the way the study was ‘hidden’ from public scrutiny sparked intense backlash from scientists. Christian Drosten, one of Germany’s leading virologists, pointed out the study’s uncertainty based on his knowledge of antibody tests and the small and localized sample size. Without a paper or data, and without being present at the press conference, his ability to filter the findings for the public was severely restricted. News outlets immediately published articles giving the public reason to doubt the severity of the virus. Protests occurred, risking further infections. Luckily for the human lives at stake, the Federal German Government stepped in and pressed State governments, such as the NRW, to follow the overall scientific evidence, not just the apparent implications of one study.

Access and transparency: key principles of open science

The Heinsberg Study should be praised for its speed. A principle of open science is to share knowledge as early as possible. Rapid, open information sharing is what led scientists to develop mass testing and simulate infection spreads as early as mid-January, allowing responsible governments to take action and save lives. But access and transparency were major failures of this study. Today there is a preprint and data are available upon request, but it took over two months after the press conference for these to materialize.

Access to results and data allows experts to check for reliability, and check if conclusions follow the results. In fairness, Streek’s claim that writing and uploading a preprint manuscript would have taken 2 to 3 weeks is valid, since current preprint servers are overloaded with submissions. Regardless, the Heinsberg Study was not reliable by definition at its first dissemination, not until other scientists checked it. Even a preprint and data sharing does not guarantee reliability. But without these things reporting a study to the media or government should be labelled as basically an op-ed news piece until it can be reviewed.

For whatever reason, if access is not possible, then scientists must take alternative measures to ensure reliability of their work. It is not only their reputations at stake. Whenever grant funding is doled out, the funder has an interest in reliable results. It is necessary to develop reliability assurance mechanisms that can be implemented in a moment’s notice; when there is no time for traditional methods. Funding bodies should form a pool of scientists who, similar to the formal peer review, pledge to be available for rapid assessments. Anyone who applies for and implements third-party funded projects knows that such committees are often necessary, in particular, to check ethical practices. This is the type of institutionalized open science that is necessary, because open science practices of researchers alone cannot change all of science.

Science by press conference: The limits of open science

When politics and the media are involved, open science often fades into background noise. The Heinsberg Study may have been politically motivated because Governor Laschet of the NRW had formed a committee of ‘experts’ to lead reopening efforts shortly after approval of the Study. Research is never devoid of motives, even independent scientific inquiries. But contract research is especially dangerous, something we know from private companies or political parties funding their own research agendas. What is essential, and consistent with open science ethics, is that it is a research contract, not a results contract. It is perhaps no coincidence that Streek was selected to conduct the Heinsberg Study, as he was a strong sceptic of the alleged severity of the virus as early as January.

What is essential, and consistent with open science ethics, is that it is a research contract, not a results contract.

The Heinsberg Study is a classic example of science by press conference. Streek reported two seemingly contradictory claims. One was that the lockdown measures had worked to slow the spread of the virus, the other was that it would be safe to start reopening. Media outlets were free to cherrypick any statement out of context. In fact, Gérard Krause, the invited scientists who should have provided a reliability check in the press conference made clear that relying on 7 deaths to calculate a case-fatality ratio is unbelievably shaky. “If you have three, four or five deaths [instead of 7] you quickly reach values above one percent” he pointed out at minute 36 of the press conference.

Once bad science is public, the damage is done

What we know from scientific publications, is that once they are out there, it is almost impossible to contain their damages. The early ‘evidence’ that immunizations cause autism was dismissed as flawed research, but some of these studies took a very long time to retract the papers, and antivax remains a mass global movement, one based entirely on false information. Gérard Krause and Christian Drosten criticized the study after it was presented in a press conference, but ‘after’ can be too late. Scientific discussions are now being held in social media formats accessible to everyone. This increases public uncertainty because comments might appear as scientific, when in fact they are intended to be politically persuasive.

What if Germany had followed Streek’s recommendations and opened up quickly? Seeing the horrific new mass infection rate spike to over 60 thousand cases per day in the United States provides a clear counterfactual. This is not just about ignoring science altogether, which was also the cause of the extremely high fatality rates in Sweden, but the dangers of relying on only one study without context. As humans we tend to look for evidence to justify what we already believe to be true. As scientists, we must resist this, and we cannot do this alone. A disaster is both a crisis and a potential opportunity. Now is a good time to reevaluate the future of scientific dissemination. Thus, having a review board ready to quickly evaluate fast science like the Heinsberg Study is my recommendation – one consistent with the principles of the Open Science Movement. Such a board should have been called upon before the press conference. Their role would be to check studies before they are released to the public, especially when this release is a hyped-up political and media event as opposed to a preprint which the public are unlikely to read.

This blog post summarizes my own position on this subject, but is derived from a longer position paper published in German. It benefited from input from Jo Havemann, Tamara Heck, Katja Mayer, Isabella Peters and Philipp Schrögel and a stipend provided by the Freies Wissen Fellowship.

Image 1 credit: clker-free-vector-images via pixabay

Image 2 Credit: ZDF adapted by author from this photo.