Sharing good practices in recognition of diversity in research and further development of open infrastructures for research information is crucial for advancing the assessment reform.  

GraspOS landscape survey conducted by the Federation of Finnish Learned Societies (TSV) in May 2023 provides a panorama of the state-of-the-art institutional research assessment practices in 54 organisations from 19 European countries (Figure 1). 

Questionnaire, data and results are published in the OS-aware RRA approaches landscape report submitted for the European Commission review 31 August 2023. Results of the Survey are also available as a presentation Landscape survey results: RRA practices in 54 research institutions.   

Figure 1. Map showing the distribution of GraspOS Landscape Survey respondent oragnisations across Europe. 

Organisations on the journey of change 

EU funded GraspOS consortium aims to develop, assess, and put into operation an open and trusted federated infrastructure for next generation research metrics, indicators, data, tools, services and guidance to support reforms for research assessment. The purpose of the GraspOS landscape survey, based on the CoARA Agreement on Reforming Research Assessment, is to support the development of Open Science Assessment Framework (OSAF).  

The survey results reflect practices mainly in universities and research institutions that conduct assessments at individual and institutional level. Over 70% of the responding organisations had already signed the CoARA Agreement, and one of their main motivations for reform is to align with the responsible research assessment movement and improve recognition of open science practices. 

While many organisations have already taken proactive steps towards assessment reform, majority of organisations (61%) considered the complexity of the reform as a difficulty, and nearly half were concerned of increasing costs. Resistance from researchers, limited awareness of the reform and lack of evidence of its potential benefits were also reported as hinderances by multiple organisations. 

Recognizing diversity in research remains a challenge 

Many organisations face similar challenges in the recognition of the diversity of research (Figure 2). Nearly all organisations consider traditional journal articles (94%) and other scientific publications (84%). Other types of research output are much less consistently recognized: datasets, software, policy contributions, methods exhibitions, protocols, theories, strategies, algorithms, data models and workflows.  

Figure 2. Recognition of aspects of diversity among 54 organisations that participated in the GraspOS Landscape Survey. 

Open access publishing is considered by 78%, and open data sharing by 54% of respondent organisations. Over 70% of organisations already recognised industry-academia cooperation, training, mentoring and supervision of PhD students as well as teaching, that were mentioned by over 70% of the organisations. Many other open science practices as well as activities and roles are less frequently considered. 

Most organisations (over70%) also take field and career-stage into account in research assessments. Moreover, questions relating to research types, interdisciplinarity, gender balance and language diversity are considered by around 60% of the organisations. However, the consideration of the EDI (Equity, Diversity, and Inclusion) dimensions remains much lower (31%). 

Balancing qualitative and quantitative evidence is crucial 

In most respondent organisations (76%), the preferred method of research assessment remains expert assessment informed with metrics. They also make some use of author-based metrics (78%) and/or metrics relating to publication venues (57%). University rankings are used in assessment by less than 10% of the organisations. However, over half of the organisations report still paying some attention to the rankings. 

Most of the organisations (82%) utilise commercial Web of Science and Scopus databases to support assessments. There also exist community-curated information sources provided by local repositories and the Current Research Information Systems (CRIS), and 76% of the organisations use these as well. These systems, however, require development for them to properly support the recognition of the diverse range of scientific contributions. 

The survey results suggest that the approaches of including qualitative input in assessment differ significantly between organisations. 57% report using some form of self-assessment and 43% a structured CV. Narrative, competency, or evidence-based CV’s; impact stories, and case narratives are still utilised in less than one third of the organisations. While consistent effort should be made to inform assessments with qualitative input, there also is an urgent need to develop understanding of what responsible use of metrics means in different assessment levels and contexts. 

Further information about the report and survey: 

The survey is conducted and analysed by a working group including Janne Pölönen, Anna-Kaisa Hyrkkänen, Dragan Ivanović and Marita Kari. In addition, Anni Rastas contributed to this news articles and presentation. 

If you have any questions you may contact: 

Anna-Kaisa Hyrkkänen 
Coordinator, Responsible Assessment & GraspOS 
Federation of Finnish Learned Societies 
anna-kaisa.hyrkkanen[at]tsv.fi 

Janne Pölönen 
Secretary General for Publication Forum 
Federation of Finnish Learned Societies 
janne.polonen[at]tsv.fi