LSE - Small Logo
LSE - Small Logo

Blog Admin

August 22nd, 2018

Developing a “responsible assessment” system to improve research impact: a case study from Catalonia

0 comments | 2 shares

Estimated reading time: 5 minutes

Blog Admin

August 22nd, 2018

Developing a “responsible assessment” system to improve research impact: a case study from Catalonia

0 comments | 2 shares

Estimated reading time: 5 minutes

Many lessons can be drawn from the burgeoning discourse on research impact assessment. Paula Adam, Maite Solans-Domènech, Núria Radó-Trilla, Toni Dedeu and Albert Barberà explain how they have applied these to the design of a health research assessment system in Catalonia, one that helps to embed impact from the early stages of a research programme. Emphasising the importance of values, leadership, culture, and empowerment, the SARIS system is focused on all research stakeholders and promotes the idea of “responsible” assessment, underpinned by openness, analysis, and engagement.

For research impact “activists” interested in designing a research assessment system that helps to embed impact from the early stages of a research programme, there are many lessons to be drawn from recent international research impact assessment exercises and experienced practitioners in the field. In our case, the International School on Research Impact Assessment (ISRIA) has been a fabulous source of such lessons, as has the available literature on the topic, including practical suggestions for implementation.

For instance, evidence suggests there are a number of enablers and facilitators of research impact; enablers being factors that are essential, while facilitators are factors that make things easier but are not essential. Our own conclusion, drawn from a review of impact studies across a number of different scientific areas, is that the enablers are: the values, leadership, and culture of the key stakeholders; and empowering research stakeholders, whether that is the researchers themselves or other key stakeholders such as healthcare professionals, clinicians, patients, or policymakers. Among the key facilitators are an organisational strategy, funding schemes, departmental organisation, internal services, collaborations, and openness. Additionally, to achieve impact both individuals and organisations must have an effective and comprehensive approach to engagement and communication with the research stakeholders and wider society.

Having learned this, we soon began to wonder how these lessons could be embedded in an assessment system before impact occurs. Can an assessment system actively foster impact? As practitioners at a Catalan research assessment agency, AQuAS, we have considered this question thoughtfully. The Department of Health of the Government of Catalonia has commissioned AQuAS to design and implement an ongoing – including beyond the end of the grant period – impact assessment system in the area of health sciences. Funded through the Department’s PERIS programme, the system we have developed is the Health Research and Innovation Assessment System (SARIS). The primary aim of SARIS is to evaluate the health research promoted under the PERIS programme, 2016-2020.

The SARIS assessment model goes beyond mere descriptive assessment. It allows for improvements across the roll-out phases of PERIS with the involvement of PERIS agents. For this reason we refer to it as “transformational assessment”. We also consider SARIS to be an example of responsible assessment, as it includes key elements from the Rome Declaration on responsible research: governance, engagement, and openness.

Because of the aforementioned importance of researchers’ (and other stakeholders’) culture, values, and leadership, SARIS is very much based around the stakeholders (by which we mean anyone who can add value to or gain value from research on an improved, more sustainable health system). SARIS aims to understand stakeholders’ incentives, motivations, engagements, and needs, with a view to providing full support.

The current incentives system encourages researchers to focus only on “excellence” in building their careers – to gain prestige and have success in grant applications; the famous “publish or perish” leitmotif. In many countries, including Spain and Catalonia, career schemes are not set up to drive research to influence, change, or otherwise affect the non-academic world. Yet things are changing from the bottom up, as researchers become more motivated to pursue a different scientific career, and concomitantly from the top down, as European research programmes increasingly require projects to demonstrate an impact on society. Therefore, beyond the need for PERIS to demonstrate its effect on society, researchers also need support as they adapt to the demands of future evaluation systems which are more balanced between academic excellence and demonstrable social impact.

SARIS has three pillars, the fundamental concepts underpinning the assessment system, the interaction of which is hoped will bring about positive impacts on society. These three pillars are:

  • Accountability – this can be interpreted in different ways according to context. In some countries, accountability is understood as the control and regulatory processes required by the funder itself (the government in this case). This is different from the Anglo-Saxon interpretation of accountability that refers to a responsibility to the taxpayers who indirectly fund research. In the SARIS context, while the public funder exerts a strong control on the use of the resources, this is purely administrative. Whether this use is interpreted as “responsible” is an open question. The accountability dimension of SARIS is different as it is oriented to the research system decision-makers and ultimately to society.
  • Analysis – this complements the accountability pillar and is an ongoing process which helps to identify potential limitations of the PERIS programme, in order to improve future iterations of the funding scheme and subsequent decisions taken. Examples of issues that have already been identified are the length of grant (is there enough time to reach the objectives?), appropriateness of the assessment criteria, the grant’s accessibility criteria, quality of funded and non-funded proposals, priority-setting, etc. Some analyses may include aspects related to “responsible research” concepts, like an analysis of gender balance in leadership roles and potential barriers or effective research engagement strategies.
  • Engagement – together with the analysis, this should lead to improved impact of PERIS. Moreover, here is where lessons from research impact assessment are applied in practice. SARIS aims to support and promote leadership among PERIS beneficiaries and researchers in general by reinforcing their abilities to communicate and engage with stakeholders and patients, and also by creating the context in which this engagement and communication can happen. Since PERIS began, the first efforts of the engagement pillar have been devoted to the community of PERIS beneficiaries – specifically the grantees of projects on nursing.

Although SARIS is a new research assessment system, having been implemented only one year ago, the interactions between SARIS and its relevant audiences (which can be seen underneath the different pillars in Figure 1) have steadily increased over time. The ultimate goal is to build a system – within and for the PERIS programme stakeholders – that, with the interaction of the three pillars, will ensure the research promoted by the Catalan Department of Health has more of an impact on society.

Engagement with the five editions of the International School on Research Impact Assessment (ISRIA) has proved a great source of learning for the authors. Impact Blog readers can freely access ISRIA teaching resources, available under a Creative Commons license. Readers can also use the international guidelines for an effective process of research impact assessment.

More information about the authors’ work can be found on AQuAS website.

Featured image credit: Tarragona, Spain, by Pau Sayrol, via Unsplash (licensed under a CC0 1.0 license).

Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.

About the authors

Paula Adam is Head of Research at the Agency for Health Quality and Assessment of Catalonia (AQuAS) and responsible for SARIS. She leads the ISOR Group, which carries out studies on the impact of health research in Spain. Paula also coordinates a commission on the accreditation and assessment of the Spanish health research institutes. She is a co-founder of ISRIA was co-director of the School when it was launched in Barcelona, Spain in 2013. Paula holds a PhD in economics from the European University Institute in Florence, Italy.

Maite Solans-Domènech has a Masters in public health and is currently a research leader at the Agency for Health Quality and Assessment of Catalonia (AQuAS). In AQuAS, she is a member of the ISOR Group and responsible for the Accountability and Analysis pillars of SARIS. Maite has participated in different projects on evidence-based medicine, the development of indicators, and on patient priority-setting.

Núria Radó-Trilla holds a PhD in biomedicine from Pompeu Fabra University (Barcelona) and MSc in science communication from Imperial College London. She is currently a researcher at the Agency for Health Quality and Assessment of Catalonia. Nuria’s work in AQuAS focuses on assessment and she is responsible for the Engagement pillar of SARIS.

 

Toni Dedeu is director of the Agency for Health Quality and Assessment of Catalonia (AQuAS).

 

 

Albert Barberà is General Director of Research and Innovation in the Department of Health of the Government of Catalonia. He holds a PhD in chemistry (biochemistry speciality) from Barcelona University and MSc in pharmacy. He has an extensive experience in the management of research and innovation; as well as in the definition, implementation, and evaluation of institutional and governmental strategies and policies to promote research and innovation.

Print Friendly, PDF & Email

About the author

Blog Admin

Posted In: Impact | Research evaluation | Research policy

Leave a Reply

Your email address will not be published. Required fields are marked *