Towards a reform of the research assessment system in Europe

The European Commission (EC) has taken significant strides towards reforming research assessment practices, aiming to enhance the diversity, transparency, and quality of research assessment across the European Union. 

Central to the reform efforts is the European Open Science Cloud (EOSC), which provides a federated infrastructure designed to facilitate access to data and the tracking of scientific information. This infrastructure connects various research organisations, enabling them to share and access high-quality data, metadata, services, and tools, thereby enhancing research assessment practices. The EC supports these efforts through dedicated funding for Horizon Europe INFRAEOSC projects, aimed at enabling an operational, open, and FAIR (Findable, Accessible, Interoperable, and Reusable) EOSC ecosystem.

In April 2024, following its signing of the Agreement on Reforming Research Assessment (ARRA), the EC published its Action Plan to implement the ten commitments outlined in the Agreement which includes a series of measures that the EC plans to carry out to further support and advance the reform.

The EC Report on Research Assessment was published in the same month, and addresses the contributions of Horizon 2020 and Horizon Europe projects to the reform of research assessment practices, following the ARRA principles and commitments.

In both EC documents, the Horizon Europe INFRAEOSC projects GraspOS and SciLake were jointly recognised as key contributors to the reform. This collaborative blog post by SciLake and GraspOS provides an overview of the aims of each project and presents their specific contributions to the reform in the areas recognised by the EC.  

GraspOS: Next Generation Research Assessment to Promote Open Science 

SciLake: Democratising and making sense out of heterogeneous scholarly content

GraspOS and Scilake emerge as key players in the research assessment reform 

The EC Report on Research Assessment identifies four clusters of activities from Horizon 2020 and Horizon Europe projects, including direct and indirect relevant contributions, from both planned and completed activities. Below, we highlight the clusters and areas where GraspOS and SciLake support the reform as indicated by the EC. 

Cluster 1: strengthen the evidence base in which to ground systemic reform and organisational change. 

• The GraspOS landscape review of approaches, tools and services for responsible research assessment that addresses open science, 

Cluster 2: development of frameworks and models to inform organisational and system-level practices

• GraspOS is developing and piloting an Open Science-aware responsible research assessment framework . 

Cluster 3: contributing to the infrastructure and dedicated tools and services for research assessment, including automated tools.

• SciLake is developing data storage and AI-assisted analytic services built on customised takes on scientific merit, and AI-assisted services for automated assessment of reproducibility and replication and of scientific, societal or economic impact.
• GraspOS is working towards an open and federated infrastructure for research assessment including tools and services, in support of Open Science-aware responsible assessment approaches. 

The EC Report also highlights promising practices identified across the projects, they are reported in the table below. 

Current progress and good practices in relation to the agreement on reforming research assessment

The identification of promising aspects and practices are also considered in relation to their level of alignment with the principles and commitments of the ARRA, and more broadly with RRI principles. Specifically, the ongoing work of GraspOS and SciLake projects is mentioned in four principles of the reform, with explicit reference to how the projects outputs under development address each principle.  

Independence and transparency

The ARRA emphasises the principles of independence and transparency in data, infrastructure, and criteria for research assessment, as well as in the analysis of research impact and transparency of indicators.
SciLake aligns with these principles by developing a service for calculating indicators for a variety of research products which will be built on transparency, the project will also provide open-source codes ensuring transparency in indicator calculations. 

Recognising impacts

The ARRA recognises the importance of contributions that advance knowledge and the potential impacts of research results at scientific, technological and/or societal level.
SciLake addresses this principle through the development of a service to manage heterogeneous scientific content with the aim to support the creation of new methodologies  for multi-perspective impact assessment, providing customizable strategies. Specifically, the project will create an impact-based discovery service which calculates impact indicators for research outputs such as publications, datasets, and software to analyse scientific, societal, and economic impact. 

Recognition of the diversity of research activities and outputs

The recognition of a wide range of research activities and practices, including peer review, PhD supervision, leadership roles, science communication, and knowledge valorization, is a central principle in the ARRA. It also includes valuing a variety of outputs such as scientific publications, data, software, algorithms, and policy contributions, and includes rewarding open science practices.
SciLake project aims to provide a technical solution which supports the assessment of a broader range of research outputs. The project will provide tools for assessing the impact and reproducibility of diverse outputs, including datasets, workflows, and protocols.
GraspOS project is developing an open and federated infrastructure for Open Science-aware, responsible research assessment. A landscape report identifies current open science assessment practices, barriers, and priorities, and highlights the need for holistic recognition of research, education, and other activities.

Criteria and processes that respect the variety of disciplines, research types and career stages

In the ARRA, two principles underline the need for assessment criteria and practices that reflect the diversity of research, including the recognition of interdisciplinarity and various research roles, both within and outside academia.
Preparatory work in the GraspOS project has indicated that best practices need to be contextualised to each assessment event and that identifying universal best practices is therefore not feasible. To address this, the project aims to develop multiple assessment portfolios, including tailorable dashboard services for funders, institutions, research teams and disciplines, along with templates for qualitative and quantitative indicators.

Conclusions

By leveraging the federated infrastructure of EOSC and supporting key projects such as GraspOS and SciLake, the EC is fostering an environment where data sharing and collaboration are rewarded, ultimately driving the advancement of Open Science and enhancing the impact and credibility of research across the European Union.

THIS ARTICLE WAS JOINTLY WRITTEN BY LOTTIE PROVOST AND STEFANIA AMODEO.