In this edition, Laura Himanen from CSC - IT Centre for Open Science answered our questions regarding the pilot focused on Assessing openness in research activities at national level in Finland.

Can you tell us about the pilot you are WORKING ON in GraspOS?

Research.fi is a service offered by the Finnish Ministry of Education and Culture that collects and disseminates information on research conducted in Finland. Its purpose is to provide an overall picture and a comprehensive information base of the research produced in Finland. Research.fi improves the location of information and experts on research and increases the visibility and societal impact of Finnish research. At the moment, Research.fi contains information on the Finnish research system, publications by Finnish organisations, projects funded by public and private research funders, information on researchers operating in Finland and their research activities, and statistical information on the development of research resources and impact. 

Why was this particular pilot selected and how does it contribute to the co-development of an Open Science-aware Responsible Research Assessment system within funding agencies and national stakeholders operating infrastructure coordination in Finland?

Research.fi is a national Current Research Information System (CRIS), and provides GraspOS an opportunity to pilot more Open Science-aware ways of looking at research activities on a national, organisational, sectoral and individual level. As part of our pilot we will also include the views and experiences of national research funders in terms of their needs for more Open Science-aware evaluation.  

Why do you think this pilot is important for the broader Open Science context?

"considering all disciplines as well as sectoral differences is necessary, and this requires thinking about Open Science in a very broad sense"

When dealing with a national CRIS, considering all disciplines as well as sectoral differences is necessary, and this requires thinking about Open Science in a very broad sense. Also, Research.fi not being an evaluation system or a tool, the questions we seek answers to are slightly different, for example, which aspects or elements of Open Science should be shown on a national, organisational, sectoral or individual level, and why - in line with the purpose of Research.fi.

How will this pilot contribute to the main aims of the project? 

Research.fi will adopt and adapt the Openness Profile to its researchers’ profile which will provide feedback from a number of individuals from different contexts within the Finnish research community on the appropriateness of different types of activities as well as indicators and metrics. We will enrich metadata using the tools and services developed in GraspOS to support the improvement of indicators and metrics in terms of quality, scope and provenance. We will also co-shape API’s of the Federated Open Metrics Infrastructure to populate Research.fi. 

Who are the main actors involved and why are they important within the Open Science and Responsible Research Assessment context of the pilot? 

Internally, at CSC, the people who are responsible for the maintaining and developing of Research.fi. External stakeholders are represented by the Steering group. The Steering group is comprised of representatives of Finnish research performing, funding and supporting organisations. 

One last question: How aware of Open Science and Responsible Research Assessment is the sector the pilot is taking place in? 

This pilot operates in the wider context of research information management, and considering how research information management plays a crucial role in research assessment, it is fairly surprising how little the documents guiding the reform on research assessment (i.e. Agreement on reforming research assessment, DORA, Metric Tide, Leiden Manifesto) have to say about it.  

Responsible research assessment principles emphasize the importance of recognizing diversity in research activities and practices, as well as prioritizing qualitative assessment and avoiding inappropriate use of metrics and indicators. Unfortunately they offer very little guidance on how to collect new types of information to support the assessment of diverse activities and practices, even less on how to utilize new types of information in assessments. So within this sector there is a lot of development work to be done in terms of defining what these new types of activities and practices could be, how can information be collected on these new types of activities and practices in a reliable way, and finally, how can new types of information be stored and utilized in a meaningful way.

Thank you, Laura!

More about the team working on the pilot
Interview: Laura Himanen, This email address is being protected from spambots. You need JavaScript enabled to view it.
CSC team for research.fi pilot: Maari Alanko, Iiris Ruoho

This time, it is Fotis Mystakopoulos, Project Policy Officer at OPERAS AISBL, who answered our questions on the pilot focused on Assessment criteria for the Social Sciences and Humanities. 

OPERAS AISBL is a non-profit organization that supports the OPERAS Research Infrastructure that coordinates and federates resources in Europe to address the scholarly communication needs of European researchers in the Social Sciences and Humanities.

Fotis, Can you tell us more about the Pilot you are working on ? 

The OPERAS Pilot is a thematic pilot focusing on the challenges the Social Sciences and Humanities (SSH) are facing with regards, of course, to research assessment, especially considering the Open Access (OA) publishing perspective. The objective is to understand what criteria are relevant for research assessment in SSH, what digital platforms are currently in use, and how we can enrich this ecosystem. There are multiple contexts to consider, and we would like to specifically investigate the inclusion of monographs and further understand OA publishing models and what they can contribute to research assessment

how WILL THIS PILOT contribute to the co-development of an Open Science-aware Responsible Research assessment system FOR thematic disciplines, such as the social sciences and humanities? 

It is well documented that SSH is one of areas that requires significant work on understanding and adapting research assessment practices to the needs of this thematic area. OPERAS is becoming established in the publishing landscape as the research infrastructure of choice. Our focus is the SSH, we have created a participatory governance structure where organisations and individuals ensure that the decision-making of OPERAS activities benefits SSH OA in general, which should enable us to bring further change to the wider sector. Through our expertise in publishing and services in development such as GoTriple, PRISM, Metrics and Diamond capacity hub, our involvement in the Open Access Book Usage Trust, we can bring a unique perspective from actors and services of the SSH field. Therefore, co-development is built in the way we operate, through the involvement of many actors in this thematic area. 

Why do you think this study is important for the broader Open Science context?

Open Science encourages us to publish more than journal articles, and research assessment could incentivise further uptake of Open Science practices. Research assessment currently, as we understand it, is measured disproportionately by quantitative indicators that look at impact through hard numbers and citations primarily from journal publications. While it is not possible to move away from this model completely, the necessity is to bring into the conversation the qualitative perspective of measuring and evaluating research and researchers. Qualitative criteria should then be included in research assessment processes, which is not yet a reality.

How will this study contribute to the main aims of the project? 

By presenting the challenges of SSH research assessment with our stakeholders and communicating the potential solutions for monitoring open science. It is important to identify the various tools at our disposal to enact change, and OPERAS will demonstrate a roadmap and potential solutions of using different indicators than the traditional ones of H-Index, and the big databases: SCOPUS and Web of Science. The purpose is to identify further data points and metrics to include in research assessment but also ways to assess those data. 

Who are the main stakeholders involved, and why are they important within the Open Science and Responsible Research Assessment context of the pilot? 

OPERAS is a Research Infrastructure with an extensive reach and established partnerships across the European Union. The governing structure of OPERAS involves organisations and individuals from 16 countries while our membership is even wider. From each country, we have national representatives that could reach out to various institutions locally and through participating in our executive assembly, or scientific advisory committee, we have a bi-directional flow of information. Another key actor is the Coalition for Advancing Research Assessment (CoARA). CoARA is coordinating the latest of a series of initiatives that want to implement Responsible Research Assessment (RRA) and as such it will coordinate many working groups, and national chapters and this increases the chances of embedding RRA practices in different contexts. At OPERAS we are involved in three of these working groups, investigating research assessment from different perspectives: multilingualism and language biases; recognizing and rewarding peer-review; and the working group that will engage with Open Infrastructures, which could be considered the closest link to GraspOS. 

One last question about the OPERAS Pilot: What are some of the challenges you foresee and how do you plan to address them?

The biggest challenge is to get consensus on what constitutes a solution for SSH, and how do we differentiate the value of SSH research to the STEM disciplines. The challenges are clear:

• • Monographs are important to SSH. So how do we measure impact compared to the traditional journals that are dominating research assessment at the moment?
  • Multilingualism: how do we embrace the various linguistic traditions when a publication that is not in English may not receive the same attention as an output written in English?
  • What does that mean for dissemination and impact, and how do we assess the quality, excellence, and impact of this output?
  • How do we measure and how do we make space for this type of resource?

These are fundamental questions that guide our journey. I already suggested the variety of stakeholders and the range of backgrounds involved, could prove the biggest challenge to overcome, because of the situational differences for each stakeholder. One way to address this issue is through a consultation process via the pilot and identifying a common platform whereby we can agree at least in what is similar for researchers in different countries and contexts (method, epistemology), and produce a set of ground-rules to work with. 

Thank you Fotis!

Information on the team working on this pilot:
Fotis Mystakopoulos, Project Policy Officer, OPERAS AISBL

In this edition of the Inside stories from the Pilot studies, we welcomed Andrea Mannocci (CNR-ISTI) and Fabrizio Pecoraro (CNR-IRPPS) to discuss about the pilot study taking place at the Italian National Research Council (CNR).  

.       .   

Andrea Mannocci              Fabrizio Pecoraro 

Hello Andrea and Fabrizio, can you tell us more about the objectives of this pilot study?

In 2022, the National Research Council of Italy (CNR) approved the "Relaunch Plan" which includes a reform of the research assessment system. In November 2022, CNR signed the CoARA agreement (ARRA), and as per its declared commitment, the whole assessment design for the career progressions of CNR researchers and technologists is deemed to change. 

The 2023 competitive call for career progressions (i.e., the latest one) embodies some core principles promoted by the ARRA and the evaluation procedure. As of June 2024, over 2500 researchers and 500 technologists across, respectively, 35 and 10 different competition areas, applied to one specific area and are still under evaluation. In each area, the evaluation is run in autonomy by a dedicated committee, which has the mandate to acquire the generic guidelines provided by the main call, translate them by instantiating the evaluation criteria valid for that research area only, and scrutinise the applications received. Similar criteria were applied in a career progression competition carried out in 2020, therefore, before the CNR signed the agreement.  

The pilot aims to analyse both the criteria applied in the 2020 selection and the ones instantiated in the 2023 competitive call, in order to capture if, and to what extent, the latter selection takes a step away from the former in applying the principles of the CoARA agreement. Moreover, once the competitive selection is over, the pilot would also gather feedback about the new procedure from both the evaluators and the evaluands on the basis of a survey.  

Additionally, from an infrastructural perspective, the pilot also aims to benchmark the suitable services and data sources federated in the GraspOS infrastructure and assess how they could potentially benefit a future hypothetical CNR-like assessment. Starting from the latest competitive call available, the requirements the candidates are asked to fulfil, and the criteria instantiated by the committees, the pilot will provide feedback both to GraspOS service and data providers, and the CNR offices in charge of the assessment design.

why was this pilot study chosen? 

Italian RPOs are generally assessed by the public agency ANVUR, which is in charge of several evaluation processes. These comprehend the periodic assessment of VQR (Evaluation of Research Quality) of RPOs (like the REF in the UK), and the National Scientific Habilitation (ASN) for teaching at the university. This evaluation is limited to the scientific production in terms of 2 to 4 research products (i.e., mainly full papers published in indexed scientific journals) published in a given period of time (i.e., the last one would consider the period 2020-2024) and also does not take into account the plurality of research products. Therefore, in our view, focusing the pilot on the production of a dashboard monitoring Open Science production and relevant indicators would be non-significant at an institutional level.

Conversely, CNR autonomously evaluates its research staff members whenever a competitive call for career progression is open. Similarly, CNR evaluates candidates (potentially external) when hiring with public competitive calls. Having signed the ARRA, it is key to understand how the institution is (and intends to) comply and fulfil the commitment.

Which aspects of open science are you looking into in relation to responsible research assessment? 

The main body of the 2023 competitive calls focuses on a diversity of contributions and careers and a true valorisation of multidisciplinarity of research products and CV titles. Candidates are allowed to compile an assessment portfolio collating research products and CV titles drawn from a fixed yet extensive selection of admitted contributions. For each, the impact is asked to be declined across manifold expressions beyond cold quantitative indicators availing of narratives.

What stage is the pilot currently in?

At the time of writing, the 2023 selection is still ongoing: all committees started to work and published the criteria they want to enforce for the selection. Some started already with the evaluation of the submitted applications and candidates are now waiting to be summoned for oral examination (expected in summer/fall 2024).

Within the pilot activities, we concluded the comparative analysis of the criteria of the two last competitive calls (2020 VS 2023), and we are in the process of designing the survey to circulate in Q1 2025 among evaluands and evaluators.

Similarly, we are currently providing feedback to service providers joining the GraspOS infrastructure and sharing details about CNR assessment design and CV templates so that services can be leveraged to design a mock-up demonstrator to be showcased to CNR offices competent for the assessment design. 

How can this pilot be relevant to enable the move towards a research assessment system that takes Open Science into account?

By scrutinising the criteria adopted by CNR committees in the current competitive call, the pilot can quantitatively and qualitatively assess the shift towards Open Science and RRA at an institutional level and frame the transition in the light of the Reform. Both the ex-post analysis and surveys can be a precious aid for the CNR offices responsible for future assessment designs and help to tweak and improve the assessment processes currently in place.

Furthermore, the pilot aims to provide a mocked-up demonstrator that leveraging GraspOS federated services and data can show how to partially automate the application process, thus moving beyond the current procedure which is entirely manual (despite being mediated by an online portal).

Thank you for sharing this story! IF you could highlight one last aspect about the pilot study before we conclude, what would it be? 

As part of the stakeholder mapping and the pilot itself, we provide a standard description of the research assessment careers advancement process. Based on UML (Unified Modelling Language) a process description and eventually its improvement can help to formally capture the points when actors (applicants, units, institutions, evaluators, evaluated) and systems interact in order to identify and extract data to define and compute indicators. This methodology can also be proposed and applied to other pilots or scenarios within the project to capture differences and similarities among research assessment procedures. 

Thank you for your time Andrea and Fabrizio!

Information on the team working on this pilot:
Andrea Mannocci (CNR-ISTI),This email address is being protected from spambots. You need JavaScript enabled to view it.
Fabrizio Pecoraro (CNR-IRPPS),This email address is being protected from spambots. You need JavaScript enabled to view it.
Francesca Di Donato (CNR-ILC), This email address is being protected from spambots. You need JavaScript enabled to view it. 
Lottie Provost (CNR-ILC),This email address is being protected from spambots. You need JavaScript enabled to view it.

We had the pleasure to exchange with Katri Rintamäki and Heikki Laitinen, both working at the University of Eastern Finland (UEF), on the pilot they are working on. In this pilot, UEF will be using novel methods to evaluate Open Science and enable responsible research assessment. 

.       

KATRI, HEIKKI, CAN YOU TELL US WHERE THE PILOT TAKES PLACE AND WHAT ARE THE MAIN OBJECTIVES?

The pilot takes place at the University of Eastern Finland (UEF), a multidisciplinary university consisting of four faculties: philosophical, science and technology, health sciences, and social sciences. The UEF scientific community comprises 16.000 students, 16.600 adult education students, and 3.200 staff members.

The UEF pilot utilises new methods to evaluate open science and enable responsible research assessment by analysing and assessing less-used indicators for open science activities. The pilot explores the possibilities to use new methods in the university’s knowledge management, impact assessment, responsible research assessment and open science evaluation. Therefore, we can say that the UEF pilot aims to find innovative ways to promote Open Science.

WHAT PROMPTED THE DEVELOPMENT OF THIS PILOT STUDY? 

University of Eastern Finland (UEF) is one of nine pilots included in GraspOS to ensure that the indicators, tools, services and infrastructure applied in assessment or monitoring are based on community needs. UEF participates in GraspOS as an affiliated entity to CSC – IT Centre for Open Science in Finland. The UEF pilot focuses on Open Science-enabled research assessment at university level. In its Strategy 2030, UEF commits to being a pioneer of Open Science and science communication. The aim is that Open Science and an open operating culture improve the quality and impact of research, and that the use of research findings by society is intensified and science reaches an increasing number of people.

Every year, the Ministry of Education and Culture in Finland collects data on scientific publications from higher education institutions and state research institutes. The publication metadata is made publicly available in the national research.fi service. Along with other Finnish universities, UEF collects annually publication data and data on research activities in its UEF Current Research Information System (CRIS) database. All the data that is collected into local CRIS databases is not exported to the national research.fi portal. The starting point for examining UEF CRIS data is information economics: whether data is sufficient; whether some data is missing; whether the collected data is utilised; what is the usability of CRIS; and how to make better use of underused data. The ideal is to find users and good use for the collected data.

CAN YOU TELL US MORE ABOUT WHICH ASPECTS OF OPEN SCIENCE YOU ARE LOOKING INTO IN RELATION TO RESPONSIBLE RESEARCH ASSESSMENT? 

We are looking especially into the aspects of open publishing and open culture. We examine how less-used publication indicators, such as multilingualism, domestic publishing, and popular publications aimed at the general public, could be used to visualise the openness and societal impact of research. Similarly, we map out what kind of research activity indicators, such as peer-reviewing, expert tasks, and citizen science activities, can be used to visualise the culture of open scholarship.

WHAT STAGE IS THE PILOT CURRENTLY IN? ARE THERE PRACTICAL APPLICATIONS OR POLICY IMPLICATIONS YOU WOULD LIKE TO SHARE WITH US?

At the moment we are analysing UEF CRIS’s less-used publication indicators and research activities in order to explore possibilities to utilise them as indicators for open science.

Widely used publication and citation rates are largely indicators for the scientific community itself. Less-used research activity indicators could highlight the social impact of research. Openness and popularisation are conducive to the use of scientific knowledge at lower levels of education, in business life, in public authorities and in political decision-making. In our pilot we aim to create recommendations on how to utilise CRIS data in a new way and what new data could be collected and utilised. We use UEF CRIS as a case example of how CRIS data could be used and developed in the future. One citizen science indicator that we would like to recommend as a new research activity indicator is the university's school collaboration and the inclusion of pupils in primary, secondary and high school in citizen science. Cooperation with different levels of education builds the future of universities.

IS THERE ANYTHING ELSE YOU WOULD LIKE TO ADD OR HIGHLIGHT ABOUT THE PILOT STUDY BEFORE WE CONCLUDE?

In this pilot study we aim to find semi-quantitative indicators that could be used in addition to narratives in open science evaluation and in responsible research assessment.

Thank you Katri and Heikki for sharing this story!

INFORMATION ON THE TEAM WORKING ON THIS PILOT:
HEIKKI LAITINEN, UNIVERSITY OF EASTERN FINLAND, This email address is being protected from spambots. You need JavaScript enabled to view it.
ANNI TARKIAINEN, UNIVERSITY OF EASTERN FINLAND, This email address is being protected from spambots. You need JavaScript enabled to view it. 
KATRI RINTAMÄKI, UNIVERSITY OF EASTERN FINLAND, This email address is being protected from spambots. You need JavaScript enabled to view it..

This month, Ana Đorđević, librarian and institutional repository manager at the University of Belgrade - Faculty of Chemistry (UNIBE), answered our questions about the GraspOS pilot study she is leading. The pilot is focused on institutional responsible research assessment at UNIBE.

Ana Đorđević

Ana, Could you please tell us about the primary aims and objectives of the pilot?

Within the institutional digital repository - Cherry, and in cooperation with University of Belgrade Computer Centre (RCUB), the aim is to develop visual models for rewarding researchers with regard to the application of Open Science. We have four goals that we plan to achieve:

1. Establishing badges for researchers who apply Green Open Access (Self-Archiving model).
2. Establishing badges for groups of researchers (departments) applying Green Open Access.
3. Establishing badges for participation in specific seminars on Open Science and RRA topics conducted by a librarian.
4. Enhancing the institutional Rulebook on Open Science to define and regulate pilot activities related to the researcher reward system, including evaluation and monitoring of researchers' activities 

why was this pilot study chosen? WHICH challenges in the current research assessment system prompted its development? 

In 2018, The National Open Science Platform defined the rules and procedures for depositing scientific results in institutional Open Access repositories. However, due to the absence of national monitoring, the assessment of researchers' adherence to Self-Archiving practices remains inconclusive.

UNIBE, as the pioneer in Serbia, established the first institutional repository within the sustainable RCUB infrastructure. Subsequently, an institutional Rulebook on Open Science was formulated. This significant step was made possible through effective communication between librarians and researchers, bolstered by the support of the Faculty board and RCUB. UNIBE continues to lead in Open Science practices, now expanding into new areas due to its immense potential. 

ARE THERE specific aspects of open science you are looking into in relation to responsible research assessment? 

We will investigate the activities of researchers and departments at UNIBE so that we can create a strong foundation for the RRA reward system. The activities mostly concern depositing scientific results in the Cherry Institutional Repository "as open as possible, as closed as necessary". We expect that the deposition of scientific results will intensify if more researchers attend seminars dedicated to OS and RRA.

WHAT stage of development is the pilot currently in?

The pilot at UNIBE is currently in its preliminary stages. A team consisting of one librarian together with a developer and three junior researchers is brainstorming ideas regarding how badges, as well as an evaluation and monitoring system will look like in the external application "Authors, Projects, Publications" (APP) of the Cherry repository.

How do you think that this pilot can be relevant to enable the move towards a research assessment system that takes Open Science into account?

With our success, we aim to set an example to all other institutions in Serbia (faculties and institutes) as an achievable model. Eventually we hope that the importance of practising Open Science and RRA will be recognized and required for further research career progression.

Is there anything else you would like to add or highlight about the pilot study before we conclude?

Thank you for including us in this wonderful project! We understand that RRA practices are in their beginning stages in Serbia, so participation in a project like this is invaluable to us as an institution.

Besides our enthusiasm to apply new reward systems, we are very glad to have the opportunity to learn from other pilots, to network and develop good practices at our Faculty and in our country. By motivating researchers to take part in all activities connected to Open Science, I believe that this would improve the overall research quality at UNIBE.

As a librarian, I find it especially important to learn from other colleagues participating in the project and present a new aspect of the librarian role at institutions (e.g. complete repository management, organising seminars and giving lectures on Open Science and RRA, etc). I am confident that we have very talented developers capable of clearly realising the reward system, as well as enthusiastic young researchers willing to take part in its testing.

Thank you Ana for sharing this story!

contact INFORMATION Of THE TEAM WORKING ON THIS PILOT:
Ana Đorđević, University of Belgrade - Faculty of Chemistry, This email address is being protected from spambots. You need JavaScript enabled to view it.

In this edition of the Inside stories from the Pilot studies, Odile Hologne, Director of INRAE’s Directorate for Open Science, shared her insights with us about the pilot supporting Open Science and Responsible Research Assessment in the Agricultural and Veterinary Sciences.  

Odile Hologne

Hello Odile, can you tell us MORE ABOUT where the pilot is taking place, and what are its main aims and objectives? 

INRAE has an Open Science policy which covers openness and FAIRness of research outputs (publications, data and code), as well as participatory research involving citizens and different kinds of stakeholders in projects to support innovation and production of new knowledge. One of the aims is to address societal challenges such as climate change, biodiversity, sustainable agri-food systems. To do so, the institute supports interdisciplinarity through specific research programs. 

The institute is also promoting a responsible research assessment for researchers to accompany their career progress, taking into account Open Science practices, interdisciplinarity, and different kinds of activities (leadership and management, training, dissemination…)

The pilot is defined in this context. We have 3 goals: 

1. Monitor the uptake of our Open Science policy,
2. Explore the feasibility of using new indicators on interdisciplinarity, support to policy or to innovation,
3. Help researchers create and enrich their openness profile and a narrative CV for assessment portfolio.

what prompted the development of this pilot study in the graspos project?

Our institute is specialised in Agricultural Sciences and already has a long history in Responsible Research Assessment which started in 2010 to better assess “applied research”.

Our Open Science policy, the Open Access repository HAL, our data repository, and the responsible use of indicators for assessment were seen as “positive assets”  to work on this pilot. 

Which are the aspects of open science that you are exploring in relation to responsible research assessment in the pilot? 

For researchers and scientists we are seeking to understand their “open science engagement”, taking into account the diversity of research outputs:

• Publications: preregistration, preprints, articles, books, open reviews, data or softwares papers that are openly accessible.
• FAIR Data and code: contribution to FAIRness, reproducibility and “as open as possible, as closed as necessary”.

What stage is the pilot currently in? Have any specific activities or milestones been achieved?

The Pilot has defined its roadmap and implementation strategy: we are building an OpenAire Gateway to aggregate data coming from different sources (publications, data, codes, grants, patents).

Beyond completeness, one of the key issues is to be able to classify outputs according to different classification (Field of Science, Science Metrix, …) and to build.

Then, we plan to have an OpenAire Monitor to analyse the uptake of our Open Science policy. We should also use Bip!Scholar to test openness profiles. 

How do you think that this pilot can be relevant to enable the move towards a research assessment system that takes Open Science into account? 

We hope to provide support to our researchers and to explain top them how they are involved in responsible research. 

Thank you for sharing this story! anything else you would like to Add about the pilot study before we conclude?

Since we all support the same goals, it is really interesting to share ideas with the GraspOS partners and to reach our goals in the project, supporting a Responsible Research Assessment to support CoARA efforts.  

Thank you for your time Odile!

Information on the team working on this pilot:
Odile hologne, director of the inrae directorate for open science, This email address is being protected from spambots. You need JavaScript enabled to view it.

We had the pleasure to discuss with Laurent Romary (Inria),Silvio Peroni (University of Bologna), and  Angelo Di Iorio (University of Bologna) about the Computer Science pilot in GraspOS. 

The pilot study covers the higher education and research community in computer science at large, and aims to analyse its scholarly specificities. In particular, the study focuses on how the community can bring specific insights in the domain of Open Science and research assessment in relation to its strong digital background.

Hello Laurent, angelo and silvio, can you tell us more about some of the challenges in the current research assessment system which prompted the development of this pilot study? 

Computer science has been a long standing domain where openness was at stake, for instance when developing open source software within a group of distributed developers or with the early development of online capabilities allowing the open dissemination of research papers and, beyond, of any kind of research production. Still, some essential characteristics of research in computer science are not taken care of in traditional cross-domain research assessment practices, in particular the importance of software as a research production that can be cited as a major achievement in a career. Another aspect which is usually missed is the importance of conferences as a major publication channel in computer science, sometimes surpassing journals in quality and fame in certain highly evolving fields such as natural language processing.

Are there specific aspects of open science which you look into in relation to responsible research assessment? 

Our priority has been to provide an exhaustive landscape analysis of open science practices in computer science and to list up all the existing initiatives or infrastructures (e.g. Software Heritage, DBLP) that can serve as a basis for a better coverage of our field from the point of view of research assessment.

What activities has the pilot carried out, and how can these support the move towards a research assessment system that takes Open Science into account? 

Our group has produced a comprehensive situation analysis in the form of a preprint report from which we identified a few experiments that could be carried out, for instance to provide a software usage dashboard for researchers and institutions. This dashboard is planned to be based upon the identification of software mentions in publications, with the perspective of eliciting both software reuse and dissemination within our community.

By focusing on software in the computer science field, which may sound like stating the obvious, we also expect to provide insights for other communities where software plays an increasing role, in particular for the production, analysis and dissemination of research data. This may contribute to a better description of a wide range of research objects in scientific assessment at large. 

Is there anything else you would like to highlight about the pilot study?

There is still a long way between setting up the kind of experiments of proofs of concepts we have in mind and integrating this in stable research assessment infrastructures. In particular, we need to see in the future how objects such as software or conferences can be fully part of open information sources such as Open Citations or OpenAlex.

Thank you for your time! 

Information on the team working on this pilot:
Laurent Romary, Director for Scientific Information and Culture, Inria
Angelo Di Iorio, Associate Professor at the University of Bologna.
Silvio Peroni, Associate Professor at the University of Bologna, Co-DIRECTOR of opencitationS

Under the spotlight in this first edition: Anestis Amanatidis, presenting the ongoing activities in the pilot focused on the Recognition and reward of open science practices in research assessment at the Copernicus Institute of Sustainable Development, Utrecht University, Netherlands. 

The Copernicus Institute of Sustainable Development at Utrecht University in the Netherlands carries out research activities across diverse disciplines to understand questions surrounding sustainable futures. How is research performed? What are the underpinning values of research which contribute to such futures? These questions are all relevant to GraspOS. In particular, the pilot will research how open science is considered in current assessment practices within the department.

Why was this particular pilot selected and how does it contribute to the co-development of an Open Science-aware Responsible Research Assessment system within universities, including departments and research groups? 

We already saw ideas of Open Science implicated in the last departmental evaluation period (2014-2020). Being right in-between the last and the new periodic evaluation, we hope to contribute to the further consideration of ideas surrounding openness and thereby contribute to the incentivisation of open science on the departmental level.

Why do you think this study is important for the broader Open Science context?

Utrecht University has - on a strategic level of the university - a programme in place to promote open science. Whilst strategically the Open Science Programme is central, the experience has been that translating it to research practice across the various faculties of the university has been challenging. Understanding the hurdles of this translation by staying close to the everyday practices of the researchers and staff is important and we hope to understand this issue better by the end of the project.

How will this study contribute to the main aims of the project? 

By being attentive to the local affordances of open scientific research and epistemic sensitivities, we account for the social as much as for the technical in the testing of tools and services developed. As such, we don’t think of the realisation of technical assets only, but also learn from the new research practices this disables and enables. 

Who are the main actors involved and why are they important within the Open Science and Responsible Research Assessment context of the pilot? 

We involve a wide range of actors, as we embarked on research in multiple empirical contexts. Key actors include the Head of Copernicus Institute of Sustainable Development in her role in the evaluation of the department, research and teaching staff who gathered to discuss matters of societal impact in a working group, as well as individuals beyond the Copernicus Institute who have a mandate over external providers of research information infrastructures.

What are some of the challenges you foresee and how do you plan to address them?

We traverse different organisational realities with the questions that we ask. Each of which have their own ideas and motivations, but also see their own challenges and risks of open science. Attending to these diverging stories in messy social contexts can become challenging in the translations to and from the ‘project level’ of GraspOS.

Thank you for sharing your story Anestis! 

More about the team working on the pilot:
This interview was carried out by Anestis Amanatidis, Utrecht University, Copernicus Institute of Sustainable Development; also on behalf of Dr. Jarno Hoekman (same department) and Prof. Dr. Carolina Castaldi (Human Geography and Spatial Planning).

This month, the Inside Stories from the Pilot Studies welcomed Ioana Spanache and Alina Irimia from UEFISCDI, the Executive Agency for Higher Education, Research and Innovation Funding in Romania.

Ioana Spanache                  Alina Irimia

Hello Ioana and Alina, CAN YOU please INTRODUCE US TO THE PILOT STUDY YOU ARE CARRYING OUT IN GrasPOS? What are your main objectives?

The focus of UEFISCDI’s pilot within the GraspOS project is to analyse, develop and test the implementation of the Openness Researcher profile within our national platforms (e.g. BrainMap), by also aligning with CoARA’s Agreement on Reforming Research Assessment, open science practices, and incorporating elements specific to narrative CVs and Openness Profile.

For this we are currently working on building a test version (a mock-up) of the openness researcher profile, based on the current BrainMap researcher profile (a platform with over 60 000 registered researchers, technicians and entrepreneurs from all over the world) to be tested in terms of content, as well as user experience, with representatives of research communities in Romania (researchers, representatives of RPOs, and other).

Specifically, the goals related to the GraspOS project are as follows:

• test the viability of the tools and services (indicator toolboxes) in accordance with the commitments of the organisation’s policies, especially related to indicators that cover more types of research products and research activities;
• analyse and test the implementation of the openness researcher profile within national platforms including user experience;
• align with CoARA’s Agreement on Reforming Research Assessment;
• experiment and do a feasibility analysis on integration with the OpenAIRE Research Graph.

IN YOUR OPINION, HOW DOES this pilot study CONTRIBUTE TO FILL IN CERTAIN gaps or challenges in the current research assessment system? 

UEFISCDI is the main funder of competitive research in Romania, and also an important policy advisor to both Ministries of Education and Research, Innovation and Digitalization, as well as a facilitator of dialogue with the research and innovation communities in Romania. 

The current assessment system tends to value especially traditional outputs like publications and citations, but other significant contributions, such as databases, software, policy briefs, and contributions to public understanding of science, are often overlooked at national level. This pilot seeks to fill this gap by expanding the definition of valuable research outputs and allowing for a broader and more personalised representation of a researcher’s impact. This aspect is directly in line with CoARA’s commitment to recognizing diverse research outputs and activities.

Also, it is worth mentioning that the initiative supports the implementation of the strategic objectives defined at national level for the transition to Open Science and a new research culture. UEFISCDI has been the leader in developing the National Open Science Strategic Framework which has at its core the White Paper of the transition to Open Science. The strategic objective 7 of the White paper talks about rethinking the current evaluation system to encompass a broader spectrum of research activities and outcomes, and take into consideration open science practices, such as early sharing of research findings, collaborative efforts, and open access to research results. 

CAN YOU TELL US MORE ABOUT Which specific aspects of open science you ARE looking into in relation to responsible research assessment? 

In the case of our pilot activity, the Openness profile will be part of the reimagined Researcher profile, as an additional layer, aggregating data from all categories of research outputs and contributions. 

Thus, we aim to bring to light a wider variety of contributions to science from categories such as research (e.g. projects, methods, data, software, publications, materials, etc.), education (e.g. courses, student/ staff supervision, mentoring, etc.), leadership, innovation, communication & engagement, and we wish to incorporate the Openness profile as a portfolio illustrating in one place a researcher’s contributions to Open Science (e.g. open datasets, method sets, software, open access publications, courses openly available etc. ). For this work in particular, we take into account CoARA’s commitments (as a signatory of the CoARA Agreement) and the OPUS Research Assessment Framework (RAF), UEFISCDI being a partner in the OPUS project and a pilot of the RAF.

HAVE SPECIFIC activities or milestones BEEN ACHIEVED?

At the beginning of summer, we launched the first version of the Researcher Openness profile. This came after a longer process of analysing research assessment processes, criteria and indicators used at national level, reviewing relevant literature, and analysing the workflows and types of data we already collect at institutional level. 

The first version of the redesigned Researcher profile, which includes a dedicated Openness profile section, a more comprehensive list of research outputs and activities and elements specific to the narrative CV, was elaborated to incorporate a broader diversity of indicators as inspired by CoARA commitments. For this, we have used as  reference the Research Assessment Framework (RAF) produced within the OPUS project. Of course, the elaboration and publication of this first version of the Researcher Profile is a major milestone in our pilot activity.

For the time being we are in the stage of receiving feedback both internally, from colleagues at UEFISCDI from different departments, and externally, from project partners and other organisations/ individuals interested in the topic. Moreover, we are also currently working on identifying potential sources of information and input that could be used for data collection purposes for the different types of indicators we are proposing. In the end we wish to be able to provide information as close as possible to the specifications needed by other departments in our organisation for implementing the Researcher profile or parts of it, in case that will be decided. 

The next steps are to design a mock up (graphical) version for the Researcher profile and then test it with researchers and research communities at national level and beyond.

How can this pilot be relevant to support the transition towards an open science-aware research assessment system?

The redesigned Researcher profile includes a dedicated Openness profile section, which aims to serve as a potential mechanism for increasing the visibility of Open Science practices, and as a result, encourage their recognition and reward. The Openness profile is envisioned as a tab or chapter in the Researcher profile that aggregates information from across all other chapters and tabs, having in addition the layer of being attributed to OS practices such as open access publications, databases, citizen science and other. 

Moreover, the built template is also intended to include detailed information about specifications needed to incorporate the Openness profile into such researcher profiles as well as into monitoring and reporting workflows. 

Thank you for your time Ioana and Alina! 

Information on the team working on this pilot:
Ioana Spanache, PhD (UEFISCDI) - This email address is being protected from spambots. You need JavaScript enabled to view it.
Alina Irimia, PhD (UEFISCDI)- This email address is being protected from spambots. You need JavaScript enabled to view it.