In this edition, Laura Himanen from CSC - IT Centre for Open Science answered our questions regarding the pilot focused on Assessing openness in research activities at national level in Finland.

Can you tell us about the pilot you are WORKING ON in GraspOS?

Research.fi is a service offered by the Finnish Ministry of Education and Culture that collects and disseminates information on research conducted in Finland. Its purpose is to provide an overall picture and a comprehensive information base of the research produced in Finland. Research.fi improves the location of information and experts on research and increases the visibility and societal impact of Finnish research. At the moment, Research.fi contains information on the Finnish research system, publications by Finnish organisations, projects funded by public and private research funders, information on researchers operating in Finland and their research activities, and statistical information on the development of research resources and impact. 

Why was this particular pilot selected and how does it contribute to the co-development of an Open Science-aware Responsible Research Assessment system within funding agencies and national stakeholders operating infrastructure coordination in Finland?

Research.fi is a national Current Research Information System (CRIS), and provides GraspOS an opportunity to pilot more Open Science-aware ways of looking at research activities on a national, organisational, sectoral and individual level. As part of our pilot we will also include the views and experiences of national research funders in terms of their needs for more Open Science-aware evaluation.  

Why do you think this pilot is important for the broader Open Science context?

"considering all disciplines as well as sectoral differences is necessary, and this requires thinking about Open Science in a very broad sense"

When dealing with a national CRIS, considering all disciplines as well as sectoral differences is necessary, and this requires thinking about Open Science in a very broad sense. Also, Research.fi not being an evaluation system or a tool, the questions we seek answers to are slightly different, for example, which aspects or elements of Open Science should be shown on a national, organisational, sectoral or individual level, and why - in line with the purpose of Research.fi.

How will this pilot contribute to the main aims of the project? 

Research.fi will adopt and adapt the Openness Profile to its researchers’ profile which will provide feedback from a number of individuals from different contexts within the Finnish research community on the appropriateness of different types of activities as well as indicators and metrics. We will enrich metadata using the tools and services developed in GraspOS to support the improvement of indicators and metrics in terms of quality, scope and provenance. We will also co-shape API’s of the Federated Open Metrics Infrastructure to populate Research.fi. 

Who are the main actors involved and why are they important within the Open Science and Responsible Research Assessment context of the pilot? 

Internally, at CSC, the people who are responsible for the maintaining and developing of Research.fi. External stakeholders are represented by the Steering group. The Steering group is comprised of representatives of Finnish research performing, funding and supporting organisations. 

One last question: How aware of Open Science and Responsible Research Assessment is the sector the pilot is taking place in? 

This pilot operates in the wider context of research information management, and considering how research information management plays a crucial role in research assessment, it is fairly surprising how little the documents guiding the reform on research assessment (i.e. Agreement on reforming research assessment, DORA, Metric Tide, Leiden Manifesto) have to say about it.  

Responsible research assessment principles emphasize the importance of recognizing diversity in research activities and practices, as well as prioritizing qualitative assessment and avoiding inappropriate use of metrics and indicators. Unfortunately they offer very little guidance on how to collect new types of information to support the assessment of diverse activities and practices, even less on how to utilize new types of information in assessments. So within this sector there is a lot of development work to be done in terms of defining what these new types of activities and practices could be, how can information be collected on these new types of activities and practices in a reliable way, and finally, how can new types of information be stored and utilized in a meaningful way.

Thank you, Laura!

More about the team working on the pilot
Interview: Laura Himanen, This email address is being protected from spambots. You need JavaScript enabled to view it.
CSC team for research.fi pilot: Maari Alanko, Iiris Ruoho

This time, it is Fotis Mystakopoulos, Project Policy Officer at OPERAS AISBL, who answered our questions on the pilot focused on Assessment criteria for the Social Sciences and Humanities. 

OPERAS AISBL is a non-profit organization that supports the OPERAS Research Infrastructure that coordinates and federates resources in Europe to address the scholarly communication needs of European researchers in the Social Sciences and Humanities.

Fotis, Can you tell us more about the Pilot you are working on ? 

The OPERAS Pilot is a thematic pilot focusing on the challenges the Social Sciences and Humanities (SSH) are facing with regards, of course, to research assessment, especially considering the Open Access (OA) publishing perspective. The objective is to understand what criteria are relevant for research assessment in SSH, what digital platforms are currently in use, and how we can enrich this ecosystem. There are multiple contexts to consider, and we would like to specifically investigate the inclusion of monographs and further understand OA publishing models and what they can contribute to research assessment

how WILL THIS PILOT contribute to the co-development of an Open Science-aware Responsible Research assessment system FOR thematic disciplines, such as the social sciences and humanities? 

It is well documented that SSH is one of areas that requires significant work on understanding and adapting research assessment practices to the needs of this thematic area. OPERAS is becoming established in the publishing landscape as the research infrastructure of choice. Our focus is the SSH, we have created a participatory governance structure where organisations and individuals ensure that the decision-making of OPERAS activities benefits SSH OA in general, which should enable us to bring further change to the wider sector. Through our expertise in publishing and services in development such as GoTriple, PRISM, Metrics and Diamond capacity hub, our involvement in the Open Access Book Usage Trust, we can bring a unique perspective from actors and services of the SSH field. Therefore, co-development is built in the way we operate, through the involvement of many actors in this thematic area. 

Why do you think this study is important for the broader Open Science context?

Open Science encourages us to publish more than journal articles, and research assessment could incentivise further uptake of Open Science practices. Research assessment currently, as we understand it, is measured disproportionately by quantitative indicators that look at impact through hard numbers and citations primarily from journal publications. While it is not possible to move away from this model completely, the necessity is to bring into the conversation the qualitative perspective of measuring and evaluating research and researchers. Qualitative criteria should then be included in research assessment processes, which is not yet a reality.

How will this study contribute to the main aims of the project? 

By presenting the challenges of SSH research assessment with our stakeholders and communicating the potential solutions for monitoring open science. It is important to identify the various tools at our disposal to enact change, and OPERAS will demonstrate a roadmap and potential solutions of using different indicators than the traditional ones of H-Index, and the big databases: SCOPUS and Web of Science. The purpose is to identify further data points and metrics to include in research assessment but also ways to assess those data. 

Who are the main stakeholders involved, and why are they important within the Open Science and Responsible Research Assessment context of the pilot? 

OPERAS is a Research Infrastructure with an extensive reach and established partnerships across the European Union. The governing structure of OPERAS involves organisations and individuals from 16 countries while our membership is even wider. From each country, we have national representatives that could reach out to various institutions locally and through participating in our executive assembly, or scientific advisory committee, we have a bi-directional flow of information. Another key actor is the Coalition for Advancing Research Assessment (CoARA). CoARA is coordinating the latest of a series of initiatives that want to implement Responsible Research Assessment (RRA) and as such it will coordinate many working groups, and national chapters and this increases the chances of embedding RRA practices in different contexts. At OPERAS we are involved in three of these working groups, investigating research assessment from different perspectives: multilingualism and language biases; recognizing and rewarding peer-review; and the working group that will engage with Open Infrastructures, which could be considered the closest link to GraspOS. 

One last question about the OPERAS Pilot: What are some of the challenges you foresee and how do you plan to address them?

The biggest challenge is to get consensus on what constitutes a solution for SSH, and how do we differentiate the value of SSH research to the STEM disciplines. The challenges are clear:

• • Monographs are important to SSH. So how do we measure impact compared to the traditional journals that are dominating research assessment at the moment?
  • Multilingualism: how do we embrace the various linguistic traditions when a publication that is not in English may not receive the same attention as an output written in English?
  • What does that mean for dissemination and impact, and how do we assess the quality, excellence, and impact of this output?
  • How do we measure and how do we make space for this type of resource?

These are fundamental questions that guide our journey. I already suggested the variety of stakeholders and the range of backgrounds involved, could prove the biggest challenge to overcome, because of the situational differences for each stakeholder. One way to address this issue is through a consultation process via the pilot and identifying a common platform whereby we can agree at least in what is similar for researchers in different countries and contexts (method, epistemology), and produce a set of ground-rules to work with. 

Thank you Fotis!

Information on the team working on this pilot:
Fotis Mystakopoulos, Project Policy Officer, OPERAS AISBL

We had the pleasure to exchange with Katri Rintamäki and Heikki Laitinen, both working at the University of Eastern Finland (UEF), on the pilot they are working on. In this pilot, UEF will be using novel methods to evaluate Open Science and enable responsible research assessment. 

.       

KATRI, HEIKKI, CAN YOU TELL US WHERE THE PILOT TAKES PLACE AND WHAT ARE THE MAIN OBJECTIVES?

The pilot takes place at the University of Eastern Finland (UEF), a multidisciplinary university consisting of four faculties: philosophical, science and technology, health sciences, and social sciences. The UEF scientific community comprises 16.000 students, 16.600 adult education students, and 3.200 staff members.

The UEF pilot utilises new methods to evaluate open science and enable responsible research assessment by analysing and assessing less-used indicators for open science activities. The pilot explores the possibilities to use new methods in the university’s knowledge management, impact assessment, responsible research assessment and open science evaluation. Therefore, we can say that the UEF pilot aims to find innovative ways to promote Open Science.

WHAT PROMPTED THE DEVELOPMENT OF THIS PILOT STUDY? 

University of Eastern Finland (UEF) is one of nine pilots included in GraspOS to ensure that the indicators, tools, services and infrastructure applied in assessment or monitoring are based on community needs. UEF participates in GraspOS as an affiliated entity to CSC – IT Centre for Open Science in Finland. The UEF pilot focuses on Open Science-enabled research assessment at university level. In its Strategy 2030, UEF commits to being a pioneer of Open Science and science communication. The aim is that Open Science and an open operating culture improve the quality and impact of research, and that the use of research findings by society is intensified and science reaches an increasing number of people.

Every year, the Ministry of Education and Culture in Finland collects data on scientific publications from higher education institutions and state research institutes. The publication metadata is made publicly available in the national research.fi service. Along with other Finnish universities, UEF collects annually publication data and data on research activities in its UEF Current Research Information System (CRIS) database. All the data that is collected into local CRIS databases is not exported to the national research.fi portal. The starting point for examining UEF CRIS data is information economics: whether data is sufficient; whether some data is missing; whether the collected data is utilised; what is the usability of CRIS; and how to make better use of underused data. The ideal is to find users and good use for the collected data.

CAN YOU TELL US MORE ABOUT WHICH ASPECTS OF OPEN SCIENCE YOU ARE LOOKING INTO IN RELATION TO RESPONSIBLE RESEARCH ASSESSMENT? 

We are looking especially into the aspects of open publishing and open culture. We examine how less-used publication indicators, such as multilingualism, domestic publishing, and popular publications aimed at the general public, could be used to visualise the openness and societal impact of research. Similarly, we map out what kind of research activity indicators, such as peer-reviewing, expert tasks, and citizen science activities, can be used to visualise the culture of open scholarship.

WHAT STAGE IS THE PILOT CURRENTLY IN? ARE THERE PRACTICAL APPLICATIONS OR POLICY IMPLICATIONS YOU WOULD LIKE TO SHARE WITH US?

At the moment we are analysing UEF CRIS’s less-used publication indicators and research activities in order to explore possibilities to utilise them as indicators for open science.

Widely used publication and citation rates are largely indicators for the scientific community itself. Less-used research activity indicators could highlight the social impact of research. Openness and popularisation are conducive to the use of scientific knowledge at lower levels of education, in business life, in public authorities and in political decision-making. In our pilot we aim to create recommendations on how to utilise CRIS data in a new way and what new data could be collected and utilised. We use UEF CRIS as a case example of how CRIS data could be used and developed in the future. One citizen science indicator that we would like to recommend as a new research activity indicator is the university's school collaboration and the inclusion of pupils in primary, secondary and high school in citizen science. Cooperation with different levels of education builds the future of universities.

IS THERE ANYTHING ELSE YOU WOULD LIKE TO ADD OR HIGHLIGHT ABOUT THE PILOT STUDY BEFORE WE CONCLUDE?

In this pilot study we aim to find semi-quantitative indicators that could be used in addition to narratives in open science evaluation and in responsible research assessment.

Thank you Katri and Heikki for sharing this story!

INFORMATION ON THE TEAM WORKING ON THIS PILOT:
HEIKKI LAITINEN, UNIVERSITY OF EASTERN FINLAND, This email address is being protected from spambots. You need JavaScript enabled to view it.
ANNI TARKIAINEN, UNIVERSITY OF EASTERN FINLAND, This email address is being protected from spambots. You need JavaScript enabled to view it. 
KATRI RINTAMÄKI, UNIVERSITY OF EASTERN FINLAND, This email address is being protected from spambots. You need JavaScript enabled to view it..

This month, Ana Đorđević, librarian and institutional repository manager at the University of Belgrade - Faculty of Chemistry (UNIBE), answered our questions about the GraspOS pilot study she is leading. The pilot is focused on institutional responsible research assessment at UNIBE.

Ana Đorđević

Ana, Could you please tell us about the primary aims and objectives of the pilot?

Within the institutional digital repository - Cherry, and in cooperation with University of Belgrade Computer Centre (RCUB), the aim is to develop visual models for rewarding researchers with regard to the application of Open Science. We have four goals that we plan to achieve:

1. Establishing badges for researchers who apply Green Open Access (Self-Archiving model).
2. Establishing badges for groups of researchers (departments) applying Green Open Access.
3. Establishing badges for participation in specific seminars on Open Science and RRA topics conducted by a librarian.
4. Enhancing the institutional Rulebook on Open Science to define and regulate pilot activities related to the researcher reward system, including evaluation and monitoring of researchers' activities 

why was this pilot study chosen? WHICH challenges in the current research assessment system prompted its development? 

In 2018, The National Open Science Platform defined the rules and procedures for depositing scientific results in institutional Open Access repositories. However, due to the absence of national monitoring, the assessment of researchers' adherence to Self-Archiving practices remains inconclusive.

UNIBE, as the pioneer in Serbia, established the first institutional repository within the sustainable RCUB infrastructure. Subsequently, an institutional Rulebook on Open Science was formulated. This significant step was made possible through effective communication between librarians and researchers, bolstered by the support of the Faculty board and RCUB. UNIBE continues to lead in Open Science practices, now expanding into new areas due to its immense potential. 

ARE THERE specific aspects of open science you are looking into in relation to responsible research assessment? 

We will investigate the activities of researchers and departments at UNIBE so that we can create a strong foundation for the RRA reward system. The activities mostly concern depositing scientific results in the Cherry Institutional Repository "as open as possible, as closed as necessary". We expect that the deposition of scientific results will intensify if more researchers attend seminars dedicated to OS and RRA.

WHAT stage of development is the pilot currently in?

The pilot at UNIBE is currently in its preliminary stages. A team consisting of one librarian together with a developer and three junior researchers is brainstorming ideas regarding how badges, as well as an evaluation and monitoring system will look like in the external application "Authors, Projects, Publications" (APP) of the Cherry repository.

How do you think that this pilot can be relevant to enable the move towards a research assessment system that takes Open Science into account?

With our success, we aim to set an example to all other institutions in Serbia (faculties and institutes) as an achievable model. Eventually we hope that the importance of practising Open Science and RRA will be recognized and required for further research career progression.

Is there anything else you would like to add or highlight about the pilot study before we conclude?

Thank you for including us in this wonderful project! We understand that RRA practices are in their beginning stages in Serbia, so participation in a project like this is invaluable to us as an institution.

Besides our enthusiasm to apply new reward systems, we are very glad to have the opportunity to learn from other pilots, to network and develop good practices at our Faculty and in our country. By motivating researchers to take part in all activities connected to Open Science, I believe that this would improve the overall research quality at UNIBE.

As a librarian, I find it especially important to learn from other colleagues participating in the project and present a new aspect of the librarian role at institutions (e.g. complete repository management, organising seminars and giving lectures on Open Science and RRA, etc). I am confident that we have very talented developers capable of clearly realising the reward system, as well as enthusiastic young researchers willing to take part in its testing.

Thank you Ana for sharing this story!

contact INFORMATION Of THE TEAM WORKING ON THIS PILOT:
Ana Đorđević, University of Belgrade - Faculty of Chemistry, This email address is being protected from spambots. You need JavaScript enabled to view it.

Under the spotlight in this first edition: Anestis Amanatidis, presenting the ongoing activities in the pilot focused on the Recognition and reward of open science practices in research assessment at the Copernicus Institute of Sustainable Development, Utrecht University, Netherlands. 

The Copernicus Institute of Sustainable Development at Utrecht University in the Netherlands carries out research activities across diverse disciplines to understand questions surrounding sustainable futures. How is research performed? What are the underpinning values of research which contribute to such futures? These questions are all relevant to GraspOS. In particular, the pilot will research how open science is considered in current assessment practices within the department.

Why was this particular pilot selected and how does it contribute to the co-development of an Open Science-aware Responsible Research Assessment system within universities, including departments and research groups? 

We already saw ideas of Open Science implicated in the last departmental evaluation period (2014-2020). Being right in-between the last and the new periodic evaluation, we hope to contribute to the further consideration of ideas surrounding openness and thereby contribute to the incentivisation of open science on the departmental level.

Why do you think this study is important for the broader Open Science context?

Utrecht University has - on a strategic level of the university - a programme in place to promote open science. Whilst strategically the Open Science Programme is central, the experience has been that translating it to research practice across the various faculties of the university has been challenging. Understanding the hurdles of this translation by staying close to the everyday practices of the researchers and staff is important and we hope to understand this issue better by the end of the project.

How will this study contribute to the main aims of the project? 

By being attentive to the local affordances of open scientific research and epistemic sensitivities, we account for the social as much as for the technical in the testing of tools and services developed. As such, we don’t think of the realisation of technical assets only, but also learn from the new research practices this disables and enables. 

Who are the main actors involved and why are they important within the Open Science and Responsible Research Assessment context of the pilot? 

We involve a wide range of actors, as we embarked on research in multiple empirical contexts. Key actors include the Head of Copernicus Institute of Sustainable Development in her role in the evaluation of the department, research and teaching staff who gathered to discuss matters of societal impact in a working group, as well as individuals beyond the Copernicus Institute who have a mandate over external providers of research information infrastructures.

What are some of the challenges you foresee and how do you plan to address them?

We traverse different organisational realities with the questions that we ask. Each of which have their own ideas and motivations, but also see their own challenges and risks of open science. Attending to these diverging stories in messy social contexts can become challenging in the translations to and from the ‘project level’ of GraspOS.

Thank you for sharing your story Anestis! 

More about the team working on the pilot:
This interview was carried out by Anestis Amanatidis, Utrecht University, Copernicus Institute of Sustainable Development; also on behalf of Dr. Jarno Hoekman (same department) and Prof. Dr. Carolina Castaldi (Human Geography and Spatial Planning).