THE INSIDE STORIES PROVIDE INSIGHT INTO THE GRASPOS PILOT STUDIES, PRESENTING THE AIMS AND CURRENT STATUS OF THEIR ACTIVITIES. LEARN ABOUT WHAT THE PILOTS ARE UP TO AND HOW THEY AIM TO BRING OPEN SCIENCE AND RESEARCH ASSESSMENT TOGETHER.
In this edition of the Inside stories from the Pilot studies, we welcomed Andrea Mannocci (CNR-ISTI) and Fabrizio Pecoraro (CNR-IRPPS) to discuss about the pilot study taking place at the Italian National Research Council (CNR).
. 
.
Andrea Mannocci Fabrizio Pecoraro
Hello Andrea and Fabrizio, can you tell us more about the objectives of this pilot study?
In 2022, the National Research Council of Italy (CNR) approved the "Relaunch Plan" which includes a reform of the research assessment system. In November 2022, CNR signed the CoARA agreement (ARRA), and as per its declared commitment, the whole assessment design for the career progressions of CNR researchers and technologists is deemed to change.
The 2023 competitive call for career progressions (i.e., the latest one) embodies some core principles promoted by the ARRA and the evaluation procedure. As of June 2024, over 2500 researchers and 500 technologists across, respectively, 35 and 10 different competition areas, applied to one specific area and are still under evaluation. In each area, the evaluation is run in autonomy by a dedicated committee, which has the mandate to acquire the generic guidelines provided by the main call, translate them by instantiating the evaluation criteria valid for that research area only, and scrutinise the applications received. Similar criteria were applied in a career progression competition carried out in 2020, therefore, before the CNR signed the agreement.
The pilot aims to analyse both the criteria applied in the 2020 selection and the ones instantiated in the 2023 competitive call, in order to capture if, and to what extent, the latter selection takes a step away from the former in applying the principles of the CoARA agreement. Moreover, once the competitive selection is over, the pilot would also gather feedback about the new procedure from both the evaluators and the evaluands on the basis of a survey.
Additionally, from an infrastructural perspective, the pilot also aims to benchmark the suitable services and data sources federated in the GraspOS infrastructure and assess how they could potentially benefit a future hypothetical CNR-like assessment. Starting from the latest competitive call available, the requirements the candidates are asked to fulfil, and the criteria instantiated by the committees, the pilot will provide feedback both to GraspOS service and data providers, and the CNR offices in charge of the assessment design.
why was this pilot study chosen?
Italian RPOs are generally assessed by the public agency ANVUR, which is in charge of several evaluation processes. These comprehend the periodic assessment of VQR (Evaluation of Research Quality) of RPOs (like the REF in the UK), and the National Scientific Habilitation (ASN) for teaching at the university. This evaluation is limited to the scientific production in terms of 2 to 4 research products (i.e., mainly full papers published in indexed scientific journals) published in a given period of time (i.e., the last one would consider the period 2020-2024) and also does not take into account the plurality of research products. Therefore, in our view, focusing the pilot on the production of a dashboard monitoring Open Science production and relevant indicators would be non-significant at an institutional level.
Conversely, CNR autonomously evaluates its research staff members whenever a competitive call for career progression is open. Similarly, CNR evaluates candidates (potentially external) when hiring with public competitive calls. Having signed the ARRA, it is key to understand how the institution is (and intends to) comply and fulfil the commitment.
Which aspects of open science are you looking into in relation to responsible research assessment?
The main body of the 2023 competitive calls focuses on a diversity of contributions and careers and a true valorisation of multidisciplinarity of research products and CV titles. Candidates are allowed to compile an assessment portfolio collating research products and CV titles drawn from a fixed yet extensive selection of admitted contributions. For each, the impact is asked to be declined across manifold expressions beyond cold quantitative indicators availing of narratives.
What stage is the pilot currently in?
At the time of writing, the 2023 selection is still ongoing: all committees started to work and published the criteria they want to enforce for the selection. Some started already with the evaluation of the submitted applications and candidates are now waiting to be summoned for oral examination (expected in summer/fall 2024).
Within the pilot activities, we concluded the comparative analysis of the criteria of the two last competitive calls (2020 VS 2023), and we are in the process of designing the survey to circulate in Q1 2025 among evaluands and evaluators.
Similarly, we are currently providing feedback to service providers joining the GraspOS infrastructure and sharing details about CNR assessment design and CV templates so that services can be leveraged to design a mock-up demonstrator to be showcased to CNR offices competent for the assessment design.
How can this pilot be relevant to enable the move towards a research assessment system that takes Open Science into account?
By scrutinising the criteria adopted by CNR committees in the current competitive call, the pilot can quantitatively and qualitatively assess the shift towards Open Science and RRA at an institutional level and frame the transition in the light of the Reform. Both the ex-post analysis and surveys can be a precious aid for the CNR offices responsible for future assessment designs and help to tweak and improve the assessment processes currently in place.
Furthermore, the pilot aims to provide a mocked-up demonstrator that leveraging GraspOS federated services and data can show how to partially automate the application process, thus moving beyond the current procedure which is entirely manual (despite being mediated by an online portal).
Thank you for sharing this story! IF you could highlight one last aspect about the pilot study before we conclude, what would it be?
As part of the stakeholder mapping and the pilot itself, we provide a standard description of the research assessment careers advancement process. Based on UML (Unified Modelling Language) a process description and eventually its improvement can help to formally capture the points when actors (applicants, units, institutions, evaluators, evaluated) and systems interact in order to identify and extract data to define and compute indicators. This methodology can also be proposed and applied to other pilots or scenarios within the project to capture differences and similarities among research assessment procedures.
Thank you for your time Andrea and Fabrizio!
