The allocation of research funding on the basis of peer review has recently come under scrutiny, due to the difficulty of assessing the difference between growing numbers of high quality applications. Presenting evidence from a large-scale survey of academics involved in the peer review of grant applications, James Hardcastle argues that academics largely see peer review as the best mechanism for allocating research funds, but that issues around peer review could be improved through increased recognition and support of reviewing as an essential requirement of academic life.
Recently there have been a number of calls for dramatic changes in the way research funding is allocated. From base funding to lottery funding and cash payments for reviewers, there is a pervasive feeling that there must be a better way to fund scientific progress.
To investigate researchers’ perspectives on this important topic, Publons recently conducted a new large-scale survey focusing on researchers’ experiences of peer review for competitive grants. The resulting report, Grant Review in Focus, is a follow-up to last years’ Global State of Peer Review report. It shows that the vast majority of researchers are generally supportive of peer review but accept that it is flawed, and – perhaps surprisingly – more formal recognition for their review contributions is seen as a stronger motivator than cash.
We conducted a number of in-depth interviews with global funders to understand their challenges and views on grant peer review, along with a global survey completed by nearly 5,000 researchers taken from the Publons user community. The researchers surveyed have reviewed, or applied for funds from over 800 unique funders, spread across 95 countries. This is, to our knowledge, the largest-ever survey on this topic.
What we found:
There are several similarities between journal peer review and grant peer activity. A small number of researchers are doing a disproportionate number of reviews, indicating a system under strain. This was confirmed by funders, who are finding it harder to allocate reviewers to grant applications.
Although 78% of researchers believe that peer review remains the best way to ensure that the highest quality proposals are funded, researchers are also aware of biases in the review process – particularly for early career researchers.
Our interviews with funders revealed that they spend up to 6 hours per application finding reviewers. Funders also stated that the process of finding willing reviewers, and ensuring they actually accept invitations and complete reviews, is becoming increasingly challenging. Our survey respondents estimated that they spend an average of 10 days per year on grant peer reviews.
Although a number of funders provide cash payments (or in the case of one funder, chocolates) to reviewers, when asked about what would encourage them to review nearly 90% of researchers selected more formalised recognition from either the inviting funder or their employers. Cash or in-kind payments was listed sixth out of eight choices in the list of preferred incentives.
The importance of transparency
Given the prevalence of taxpayer-funded government and charitable funding in research, there is an increased argument for transparency and accountability around the grant peer review process. Our research found considerable support for increased transparency in the grant peer review process. This support was also higher than that found amongst fellow researchers engaged in journal peer review.
Alternatives to peer-review
As the number of applications is increasing and the success rate of grant applications is falling, the amount of peer review required for ultimately unsuccessful grants has increased. This has led to increased discussions on the best way to allocate research funding, the most prominent suggestions being lottery funding and base funding for all researchers.
Most proposals for lottery funding would require some degree of triage to ensure the validity of proposals. One method would be a streamlined peer review process to check proposals to for suitability and scientific validity with accepted proposals then added to pool of candidates for the lottery. This is not dissimilar to the triage system used by megajournals, such as PLOS ONE, for journal peer review, which would reduce some of the complexity and evaluative judgements of relative merit, but would not remove the need for peer review entirely.
One paper, based on the Canadian research funding system, has suggested a base funding process. It estimated that all researchers could be given a grant of average size if the cost saving of peer review and grant preparation were passed on to researchers. Yet even in this situation, a decision is required as to who is eligible, and previous challenges about eligibility for the REF in the UK indicate that the challenge may move to the university or department level if there is pressure to restrict the number of applications.
Artificial Intelligence tools could be another way of disrupting the grant peer review process, but these hold a documented risk of replicating existing biases in the systems they are attempting to replace, driven, in part by the pre-existing biases in the training data. Our survey found that researchers believe that the current peer review system exhibits bias against ECRs, if so, this would already be present in the data, and AI might not offer much improvement.
Improvements to the system
Peer review has many benefits and is widely supported researchers, but it is not above reproach and there are several areas highlighted by our interviews with funders and researcher survey for improvement. Over 85% of researchers believed the system would be improved from better peer review training for grants; improved systems for managing peer-review; and improved communications between the various stakeholders involved. The highest ranked factor for improving the grant peer review process was greater recognition for reviews.
There is very uneven distribution of review workload within the systems. Based on self-reported estimates, five per cent of reviewers are doing 25% of the reviews, but there was less support for a formalised review quid-pro-quo where applicants for grants must also agree to review future applications.
Reviewers generally review for extrinsic reasons such as service to the field, or reciprocation for others reviewing their own grants, in addition to viewing peer review as a vital part of their role as a researcher. The extrinsic reason also ranked more highly for grant reviewers than in journal peer review.
Researchers view peer review as the best way to ensure the best research is funded, and continue to demonstrate willingness to undertake the work, but ultimately the system in which they are operating can and must be improved to ensure the long-term sustainability of science funding.
About the author
James Hardcastle is a Senior Analyst at the Web of Science Group, part of Clarivate Analytics.
Note: This article gives the personal view of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.