The third edition of Connecting the Dots brings together representatives from the European University Association (EUA), the Université de Lorraine, and the University of Belgrade - Faculty of Chemistry in a discussion on Open Science monitoring for research assessment purposes. 

Following the GraspOS Community of Practice meeting held on 22 January 2025, Ana Đorđević, Rita Morais, Biljana Kosanović, and Nicolas Fressengeas have accepted to answer our questions and share their knowledge and insights on monitoring of Open Science in their respective organisations.

Can you tell us more about your institution and how it is engaging with current efforts to reform research assessment?

 Nicolas: Université de Lorraine is a large research intensive university of more than 4000 researchers and research support staff, with an important disciplinary scope, ranging from Humanities and Social Sciences to Science, Technology and Medicine. It has been strongly engaged in advancing research assessment ever since the beginning of 2022, when it contributed to the Paris Open Science European Conference, OSEC2022, that launched the Paris Call on Research Assessment. This call contributed to the creation of CoARA, the Coalition for Advancing Research Assessment, of which Université de Lorraine naturally became a member. At the time and following the signature of the DORA, the Declaration of San Francisco on Research Assessment, it had already engaged in producing internal advice and procedures to emphasise the qualitative nature of research assessment over a purely quantitative one. As of today, it has continued the process by involving the whole university research community starting in June 2024, with the aim to publish its CoARA action plan in the first half of 2025. 

 Biljana: I’m working at the University of Belgrade Computer Centre and we developed and maintained sustainable infrastructure for 48 repositories from scientific and academic institutions in Serbia. We also developed the national CRIS eNauka (eScience). We introduce as much data on openness as possible into the national CRIS, as well as data that can be obtained by using open tools (e.g. OpenCitations). In this way, we present the existing data to the decision makers on evaluation, with the hope that they will include them in the construction of new indicators.

 Ana: I work as a librarian and manage the Cherry institutional repository at the University of Belgrade - Faculty of Chemistry. Cherry, developed by the University of Belgrade Computer Centre in 2018, was the first faculty-specific repository of its kind. In addition to managing Cherry, I also coordinate the GraspOS project at the Faculty, a project which focuses on Responsible Research Assessment - a cause which I am passionate about! As part of this initiative, we’ve integrated Cherry with infrastructures such as OpenCitations and BIP! Ranker to promote Open Science practices. We’re currently developing a new reward system to recognise and encourage researchers for good practices in Open Science and research transparency.

 Rita: The European University Association has been actively involved in research assessment reform for several years. Initially, the Association focused more closely on research assessment in the transition to Open Science. Over time, the reform of academic careers, including assessment, has become a strategic priority, as reflected in EUA’s strategic plan and its vision for universities in 2030. EUA has also been closely involved in the drafting of the Agreement on Reforming Research Assessment and in establishing CoARA. Currently, the Association coordinates the CoARA Working Group on Reforming Academic Career Assessment. At the same time, EUA continues to inform and raise awareness among its members about the reform of research assessment.

Does your institution have an established Open Science policy? If so, what are its main components? 

 Ana: Yes, we have an institutional Rulebook on Open Science at the Faculty. It is currently being revised to align with the national Open Science Platform 2.0 and our ongoing development of a new researcher reward system. The updated regulations will reflect these changes. The text of our current regulations is available here.

 Nicolas: Yes it has! It gets its inspiration from the French national plan for Open Science. As such, its main components aim to open the production of its researchers as widely as possible. The term production encompasses the researcher’s writings — papers and books — as well as the research data and software they produce. However, Université de Lorraine strongly suggests that article publication charges are not the way to go towards open access, as it is not sustainable in the long run and inherently induces conflict of interest at the publisher level. In the same spirit, the University has a cautious approach towards so-called transformative agreements and recommends the use of community-owned or controlled infrastructures for the opening of all research productions.

 Rita: The work of EUA on Open Science is framed by the Association’s Open Science Agenda, which defines three main priority areas for EUA’s work on Open Science: Open Access to scholarly outputs in a just scholarly publishing ecosystem, FAIR research data, and research assessment. On Open Access, EUA advocates for academic ownership of scholarly communication and publishing, as well as a just scholarly communication ecosystem (i.e. transparent, diverse, economically affordable and sustainable, technically interoperable, and steered by the research community). EUA calls for FAIR research data as the norm in producing and sharing scientific knowledge, for new professional profiles for data-intensive careers, and for an active engagement in EOSC. On research assessment, the Association advocates for a responsible, transparent, and sustainable research assessment system, for the inclusion of Open Science as part of research assessment practices, and assessment approaches that balance qualitative and quantitative metrics.

 Biljana: I am more involved at the national level. The Open Science Platform 2.0, the new national policy in Serbia, was approved in December 2024 and emphasises five key goals: (1) Open Access to scholarly publications; (2) availability of research data; (3) open and transparent access to research infrastructures; (4) transparency of scholarly communication and methodology, including the availability of software source code, design documentation of research hardware and other digital objects used in the analysis of research data, and (5) the development of the digital infrastructure and competencies that make it possible to achieve the goals of the policy. Its main elements are described here. 

Does your institution monitor Open Science practices for research assessment purposes? Are there specific indicators or frameworks used to measure Open Science contributions?

 Biljana: At the institutional level, the OpenAIRE Monitor Dashboard is used but only for an overall picture. At the national level, we have incorporated a similar measure of openness within eScience. Unfortunately, during the discussion on the national policy (just adopted, and mentioned above), no consensus was reached, but it was written: "The Ministry will define mechanisms for rewarding researchers who consistently follow the principles of Open Science through publishing research data, software and hardware in open access, pre-registration of research, pre-prints, open peer review, citizen science projects, development of free and open-source software and open hardware, open materials, etc. Institutions should encourage and reward Open Science practices as far as their resources allow.” It certainly shows a clear desire, but not the intention to put it into practice right now.

 Ana: We are proud to be part of the OpenAIRE Monitor Dashboard, where the main insights we gain include key metrics on our research outputs, Open Access compliance, and collaboration with international research funders. In partnership with the University of Belgrade Computer Centre and supported by the Horizon Europe GraspOS project, we are developing the first institutional researcher reward system in Serbia. This system is based on three key principles:

1. Awarding badges to researchers who share their scientific results through green open access via the institutional repository Cherry. This enhances the visibility of scientific works, especially those whose published versions are restricted due to publisher rights;

2. Awarding badges to departments that facilitate green Open Access by encouraging researchers to share their results through Cherry;

3. Awarding badges to researchers who actively participate in seminars on Open Science and Responsible Research Assessment, organised by librarians.

Through these initiatives, we aim to incentivise and recognise contributions to Open Science practices and Responsible Research Assessment.

 Rita: Since 2016, EUA has conducted several large-scale surveys among its members, gathering information on the establishment of Open Science policies and practices. The 2019 and 2020-2021 editions specifically examined Open Science in research assessment in European universities, showing that the inclusion of Open Science elements in academic assessments remained limited. More recently, findings from the CoARA Working Group on Reforming Academic Career Assessment indicate that Open Science activities are considered low or moderately important in researchers’ career evaluation, but this is expected to improve as reforms on research and academic career assessment are implemented.

 Nicolas: The answer is no. Université de Lorraine is nonetheless highly engaged in Open Science monitoring as it heads the French national project that allows to release the French Open Science monitor. It is also deeply involved in the Open Science monitoring initiative which aims, with UNESCO and the French Ministry for Higher Education and Research, to bring together Open Science monitoring initiatives worldwide. As such, it provides a detailed monitor on its own Open Science achievements, providing many indicators, among which are the Open Access rate and the proportions of papers that share their data and software.

However, the purpose of this monitoring is not directly linked to research assessment. It rather sheds light on the road to crafting and adapting evidence-based Open Science policies. It also allows showcasing the opening of science by a given laboratory or a larger research community. If these communities choose to do so, they can however ask to add this monitoring in the assessment of their research.

Initiatives like CoARA have highlighted the need to base research assessment primarily on qualitative evaluation for which peer review is central, supported by responsible use of quantitative indicators. Are there ways in which your institution promotes the responsible use of metrics in research assessment?            

 Rita: EUA is a member of CoARA and, as outlined in our Action Plan, we will continue to support our members in reflecting on institutional approaches to research assessment. We will also continue to advocate for responsible, transparent and sustainable assessment practices, raise awareness, build capacity and create support for university initiatives among the main actors (including [early-career] researchers, other research performing organisations, research funders and policy makers).

 Nicolas: This is all the purpose of our commitment to CoARA. As suggested above, we had already modified our message to research evaluators to engage them in a more qualitative process. This process will be asserted in 2025 by our CoARA action plan, the aim of which is exactly this goal.

 Biljana: There is clear dissatisfaction in the research community with the current indicators, but no clear proposals for how this could be changed. From our side (primarily  IT implementation though eNauka), we show that such measures exist, and that there is space for discussion about them.

 Ana: While we are not currently participating in the CoARA initiative, we are committed to promoting Responsible Research Assessment. We are actively exploring ways to integrate qualitative evaluation with responsible use of quantitative indicators to ensure a balanced and fair approach to research assessment within our institution.

What are the main challenges your institution faces in reforming research assessment? What specific services or infrastructures are needed to further support Open Science-aware research assessment? 

 Biljana: Serbia is a small scientific community (max. 25,000 researchers at all levels, including PhD students), and therefore all evaluation decisions are made at the national level. The institutional policies are grounded in national-level legislation. Through the funding of eNauka, the Ministry enables comprehensive and public access to all outputs produced by the scientific community. This initiative provides hope that the Ministry will be prepared to amend normative acts and legislative regulations concerning evaluation practices.

 Ana: In my perspective, a significant challenge we face is the lack of librarians at the faculties. This lack of specialised staff makes it difficult to identify and engage the right experts to organise and conduct Open Science seminars and workshops - efforts that are crucial for advancing Open Science practices within the research community. To better support Open Science-aware research assessment, it is essential that we invest in the training and engagement of more librarians and Open Science specialists, in order to build the infrastructure needed for these initiatives.

 Nicolas: Recognising the efforts that it takes to implement Open Science is only one driver, among many, of the reform. Indeed, the focus on publish or perish also hinders other research activities such as its links with society. We feel that the main challenge is the need for a coordinated global culture change. As Toma Susi puts it, Open Science needs no martyrs: any local only change is doomed to fail. This worldwide coordination is CoARA’s goal.

We feel that Open Science-aware research assessment needs to solely rely on open data: on data that can be accessed equally by assessors and assessees. This is the reason Unversité de Lorraine has redirected its data curation workforce from closed databases to the open OpenAlex. Consequently, and until OpenAlex open data can be fully trusted for this type of activity, Université de Lorraine has taken its distances with international rankings, most of which are closed-source based. This last step is crucial to unlock the possibilities of real change.

 Rita: As Nicolas pointed out, reforming research assessment requires a change in academic culture – a process that is both long and challenging. Many institutions are eager to implement reforms aligned with DORA and CoARA, and are taking important steps in that direction. There is a clear momentum for change. At the same time, there is a lot of uncertainty on how to implement reforms in practice, how these will unfold and which consequences they may have, particularly for early-career researchers. The success of reform efforts depends on multiple factors, including stakeholder engagement, available resources, academic cultures, and regulatory and policy frameworks – each of which can either facilitate or hinder reform efforts. But what we have seen, and gathered data on, is that the motivation to change, one step at a time, remains fundamental to reform progress. Peer learning and exchanges between universities, participation in platforms like CoARA and other initiatives on reforming research assessment, national and international coordination, are all needed to sustain the momentum for change in research assessment.

Thank you very much for your time Ana, Rita, Biljana, and Nicolas!  

To learn more about the role of context in Responsible Research Assessment, we invite you to take a look at the slides from the Community of Practice meeting which have been made available here.