On the occasion of its second plenary meeting in Leiden in January 2024, the GraspOS consortium celebrated the first year of the project, highlighting progress in various areas, and notably in the development of the Open Science Assessment Framework (OSAF). 

The OSAF seeks to support current and future signatories of the Agreement on Reforming Research Assessment (ARRA) in fulfilling their commitments to transforming research evaluation practices. The development of the OSAF is guided by the principles articulated in the ARRA and by the SCOPE Evaluation Framework, and takes place within GraspOS through co-creation activities involving the nine pilots of the project. 

The nine pilots are tasked with testing ways of implementing Open Science-aware research assessment in a variety of settings, ranging from funding agencies, to universities, including departments and research groups, up to thematic disciplines. The pilots serve as co-production partners in developing the OSAF. 

Operationally, GraspOS has adopted the SCOPE evaluation framework as the common approach for piloting Open Science-aware responsible research assessments. As such, each of the nine pilot assessments follows the SCOPE process in their respective projects.

In the first year of the project, three workshops were organised to engage pilot representatives in refining and contributing to the OSAF.

1. Stakeholder Mapping Exercise (Online, October 2023)
2. OSAF Method - Values Statement (In-person, November 2023)
3. Tools and Services Exploration (In-person, January 2024)

We provide a summary of each workshop below. 

Workshop 1: Stakeholder Mapping Exercise – October 2023 (Online)

The first online workshop took the form of an exercise in which pilot representatives identified the various groups of stakeholders they aimed at including in their piloting activities, using guidance from the OSAF and the SCOPE Framework. The exercise was facilitated by the use of templates (see section 6. Annexes, GraspOS Deliverable 2.2 "OSAF").

Pilots were asked to provide a graphic or illustrated representation of their stakeholders, starting with the “Start with what you value” step from the SCOPE framework.

Figure 1. SCOPE Framework.
Taken from: Himanen L, Conte E, Gauffriau M et al. The SCOPE framework – implementing the ideals of responsible research assessment [version 1; peer review: 1 approved with reservations]. F1000Research 2023, 12:1241.

It was noted that while each pilot and organisation face unique challenges, some overlaps may occur. For instance, thematic disciplines often intersect with institutional or national processes, and vice versa. 

Each pilot representative created a mapping of the target audience and key stakeholders they would need to engage with, and presented their setting to the rest of the participants. An example of the stakeholder mapping exercise is provided below, it is taken from the GraspOS pilot for the Social Sciences and Humanities (SSH). 

Figure 2: Stakeholder mapping from the SSH pilot

Workshop 2: OSAF Method and Values Statement - November 2023 (Espoo, Finland)

The second workshop took place in-person and delved deeper into integrating SCOPE principles into the OSAF. A central element of the preparatory work for the workshop was the development of a values statement by each pilot representative. 

Values statements serve as the expression of an organisation’s priorities within the context of their evaluation processes. They are structured in three distinct tiers; 

• Super-values: overarching ideals aligned with an organisation's vision
• Values: the translation of super-values into actionable concepts
• Subvalues: when values become tangible practices which are tailored to each organisation's needs

Understanding and articulating these three components is crucial for organisations who aim at building a consistent and effective evaluation framework which resonates with their guiding principles and ambitions. 

The need to ensure the diversity of research outputs, and to think about the impact and quality of research emerged as a common denominator across the value statements provided by the pilot representatives.  

Efforts were also made to improve collective understanding of terms within the OSAF, leading to the creation of a Glossary for the GraspOS Deliverable 2.2 Open Science Assessment Framework. 

Workshop 3: Tools & Services Exploration – January 2024 (Leiden)

The third workshop centred on the development and use of GraspOS tools and services for the pilots. 

The pilot representatives were split into three groups based on their pilot setting, and each attended three workshops, one for each category of services. The sessions aimed at presenting possible scenarios which could be fit for the pilots, allowing  pilot representatives to discuss the specificities of their pilot and to ask questions. 

GraspOS services are categorised as follows:

Table 1. GraspOS services by category

The openness of assessment infrastructures and tools is crucial, and GraspOS strives to provide non proprietary resources, ensuring transparent processes in research assessment exercises and allowing for data use and reuse. Many of these services are already openly available on platforms such as Zenodo or Github (Example: BIP! Ranker - GitHub; BIP! NDR workflow & Citation Classifier - Zenodo).  

Summary

The workshops highlighted the iterative and collaborative nature of OSAF development, informed by pilot experiences and the ARRA and SCOPE principles. As the project enters its second year, the pilots’ efforts will focus on community engagement, with open consultations to take place, stay tuned!

This article was written by Fotis Mystakopoulos (OPERAS) and Lottie Provost (CNR)