LSE - Small Logo
LSE - Small Logo

Blog Admin

October 31st, 2018

Rewarding peer reviewers: a problem of adverse selection?

0 comments | 4 shares

Estimated reading time: 5 minutes

Blog Admin

October 31st, 2018

Rewarding peer reviewers: a problem of adverse selection?

0 comments | 4 shares

Estimated reading time: 5 minutes

As a solution to the worsening peer reviewer shortage, many scientific journals now offer non-monetary rewards as an incentive to researchers to register to review. Marco Seeber and Monica Zaharie report on research studying the efficacy of non-monetary rewards in attracting peer reviewers, exploring whether acceptance varies according to the nature of the offer of the reward or the researcher’s gender, rank, or productivity. While it is often taken for granted that offering a reward will spur a particular behaviour, this is not necessarily true for an undertaking such as peer review, where efforts are often also driven by ethical commitment and other more intrinsic motivations.

Scientific journals face a worsening shortage of peer reviewers. To address this problem, some journals offer monetary rewards, while many more offer non-monetary rewards, such as awards for the best reviewers and public acknowledgment of a review.

Scholarly studies have mostly focused on the impact of monetary rewards, showing potential side effects such as the crowding out of intrinsic motivation. Conversely, it is often assumed non-monetary rewards avoid the drawbacks of monetary rewards, despite research on the efficacy of non-monetary rewards for peer review being limited.

Therefore, we studied the efficacy of non-monetary rewards in attracting peer reviewers through a natural experiment implemented by a Romanian business and economics journal. The journal aimed to extend its pool of referees and sent out an invitation to 1,865 economists in Romanian universities. As a reward, the journal offered an acknowledgment in the journal and the issuance of a review certificate, and explored the efficacy of four experimental settings:

  • Non-reward: no reward offered for accepting the invitation to become a reviewer
  • Engagement: reward offered for accepting the invitation to become a reviewer
  • Completion: reward offered for completing at least one review report
  • Performance: reward offered based on the quality and quantity of reviews

We explored whether acceptance to become part of the pool of reviewers varies according to the experimental settings and the scientist’s gender, academic rank, research productivity, and type of institutional affiliation (public/private).To do this we used binary multilevel multivariate regressions – a full write-up of our study has recently been published in Scientometrics.

Efficacy of non-monetary rewards in attracting reviewers

Overall, we discovered that the acceptance rate is not higher in the reward settings (i.e. engagement, completion, and performance) compared to the non-reward setting, and is much lower (60%) under the performance setting. We also found that the impact of the reward varies significantly across subgroups of reviewers. Women are more responsive than men to rewards, especially under the performance setting. Such higher responsiveness may be due to the fact that, in many contexts, women still struggle to be recognised as having equal status to men. Therefore, an award that recognises their effort or superior capabilities has a comparatively greater value to them.

Figure 1: Acceptance rate by gender and experimental setting.

Academics from private universities are much more likely to accept reviewer invitations when no reward is offered, whereas the reverse is true of scientists from public universities (except under the performance setting). As a matter of fact, publishing the name of the reviewer signals its service to the academic community, which is likely more welcome in a public institution than in a private institution, as in this context it also signals that the scientist is diverting resources from the goal of a profit-oriented organisation.

Figure 2: Acceptance rate by type of institutional affiliation (private or private) and experimental setting.

Those in academic ranks more focused on research (in Romania, associate professors) are more likely to accept invitations to become a reviewer than those in academic ranks which are relatively more focused on teaching (i.e. assistant professors and teaching assistants) and managerial tasks (i.e. full professors).

Figure 3: Acceptance rate by academic rank and experimental setting.

Finally, we found that the performance setting is particularly unattractive to highly productive scientists (with productivity being measured through a factor score combining scientists’ number of research articles, number of proceedings articles, and Hirsch index of citations, sourced from Web of Science). Arguably, this is due to highly productive scientists having strong intrinsic motivations to peer review, while the performance setting increases the pressure felt and diminishes enjoyment of the task.

Attracting reviewers: implications and possible solutions

It is often taken for granted that offering a reward spurs a particular behaviour. This is not necessarily true for efforts that are also driven by ethical commitment and pleasure (intrinsic motivation). On the contrary, rewards — particularly those contingent on performance — may unleash an adverse selection process, by attracting extrinsically motivated reviewers and discouraging those with scientifically productive, intrinsic, pro-social motivations. Given that intrinsic motivation increases curiosity and cognitive flexibility, which are pivotal for complex tasks that require deep information processing, then such a subtle selection effect can have negative consequences on a journal peer review.

In turn, to encourage tasks like peer review, the expression “pay/reward enough or don’t pay at all” seems appropriate: if the reward is sufficiently important it will have a positive effect, otherwise it will have a negative effect, because scientists will either not feel compelled by social norms any longer, or feel their genuine drive diminished. Therefore, in order to attract reviewers, journals should introduce monetary or non-monetary rewards that are sufficiently valuable for competent and motivated researchers, or provide rewards that are not framed as such, but as signs of a trustful, grateful relationship – like providing unexpected “gifts”, such as subscription discounts, and keeping reviewers updated on the fate of a manuscript – which can foster a sense of reciprocity and of being part of a community.

This blog post is based on the authors’ article, “Are non-monetary rewards effective in attracting peer reviewers? A natural experiment”, published in Scientometrics (DOI: 10.1007/s11192-018-2912-6).

Featured image credit: Open Clipart-Vectors, via Pixabay (licensed under a CC0 1.0 license).

Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics.

About the authors

Marco Seeber is postdoctoral researcher at Ghent University.

Monica Zaharie is associate professor at Babeș-Bolyai University.

Print Friendly, PDF & Email

About the author

Blog Admin

Posted In: Peer review

Leave a Reply

Your email address will not be published. Required fields are marked *