The recently submitted GraspOS Deliverable 2.4 OSAF Specifications for Pilots details the structure and main components of the Open Science Assessment Framework (OSAF).
This latest iteration of the OSAF features an extensive list of resources to support that augment each phase of the INORMS SCOPE framework, while also providing three assessment services that facilitate increased collaboration within an assessment event and increase interoperability with local CRISs (Assessment Portfolio), increased reuse of assessment protocol (Assessment Registry), and increased visibility of researchers’ contributions to Open Science (Openness Profile).
The extensive list of assessment process resources for planning and conducting research assessment, paired with three assessment services are co-developed with the GraspOS pilots. With the release of the latest iteration of the OSAF, one of the main aims of the GraspOS project comes closer to fruition.
The Deliverable 2.4 report outlines the OSAF component specifications, explains how the components interact, and provides a foundation for practical planning, testing, and refinement of OSAF in conjunction with implementation by the GraspOS pilots.
Figure 1. OSAF high-level
Taken from: GraspOS Deliverable 2.4 OSAF Specifications for Pilots, p.9
The OSAF aims to support signatories of the Agreement on Reforming Research Assessment (ARRA) in their efforts to implement the agreed commitments. To this end the framework brings together assessment processes and assessment infrastructure as integral parts of a ‘research assessment event’. The notion assessment event emphasises the time boundedness of formal research assessments.
Assessment resources
On the process side, in this latest iteration the INORMS SCOPE approach is further integrated in OSAF, thereby also providing an overarching logic for use of the individual resources. This will bring two main advantages: first, SCOPE is an increasingly established approach for implementing Responsible Research Assessment (RRA); second, it supports the combining of existing and emerging resources that enhance familiarity and usability, and avoid duplication of effort.
The resources provided come in three formats - templates, guides, and checklists - and are meant to support the design and documentation of research assessment events while also providing guidance toward operationalising RRA principles while respecting the flexibility needed to implement in local contexts. Following is a list of the OSAF assessment resources in relation to each of the five SCOPE phases.
Table 1. OSAF resources by SCOPE assessment process phases
Taken from: GraspOS Deliverable 2.4 OSAF Specifications for Pilots, p.14
A few examples of the available resources are provided below. The full list can be found in the Deliverable.
| Responsible assessments checklist | A self-evaluation resource for exploring the whole assessment process in detail. The purpose is to make sure that the principles of responsible assessment are followed in an assessment process. |
| Purpose and context statement template | A resource for designing and documenting the assessment process. The template will help consider the specific needs and requirements for evaluation events (like indicators, methods, data, tools and services) according to the purpose of evaluation (e.g., monitoring, learning and improvement, or resource allocation and career assessment) and the level of assessment (e.g., individual, unit, institution, country). |
| Indicator toolbox for assessment guide |
Depending on the purpose, context and level of analysis required, the toolbox will provide a heuristic for selecting the appropriate indicators. |
| Narrative CV template |
Building on the Résumé for Researchers template, the narrative CV template will provide prompts and definitions, and possibly a module, for recognising a broad range of research outputs, competencies, and contributions to science as well as Open Science practices. |
| Guide for evaluators and evaluands | This guide will aim to help evaluators and evaluands understand the basic framework of rules and principles of responsible research assessment according to international and national legal regulation. |
|
Guide on the diversity of OS contributions and activities |
The aim of the Guidance is to ensure that a wide range of practices and activities are considered, and that all who contributed are recognised. |
|
Data, Tools, and services guide |
The GraspOS Data, Tools, and Services guide offers guidance on the optimal and correct use of the datasets, tools, and services that are currently part of the GraspOS open and federated infrastructure. |
Assessment services
The OSAF assessment infrastructure consists of three services currently being developed to support the structured collection and organisation of research assessment information, from initial planning to the publication of assessment protocols. These are the Assessment Portfolio Registry, the Assessment Protocol Registry and the Openness Profile Registry. All three services are underpinned by the Research Activity Identifier (RAiD) and are part of the GraspOS Federated Open Infrastructure.
Figure 2: OSAF process resources and services in the GraspOS infrastructure
Taken from: GraspOS Deliverable 2.4 OSAF Specifications for Pilots, p.9
Assessment Portfolio Registry
Currently in development, in collaboration with the FAIRCORE4EOSC project, an assessment portfolio is a collaborative, multi-actor digital object that brings together the key information about assessment planning that informs the assessment design. The assessment portfolio also facilitates the delivery of content to downstream analytics (via API) and the documentation of the assessment outcomes. As this guide is relevant in other assessment phases, and its role is different in each phase, its specific role will be described in each relevant phase.
Assessment Protocol Registry
The Assessment Protocols Registry is designed to register and publish assessment protocols after the completion of an assessment event. These registered protocols would ideally include documentation of the values, purpose and contextual factors related to the assessment, as well as the data sources and indicators used in the assessment. Individual identities and specific evidence used in the assessment are not included. Register an assessment from the GraspOS service point.
Openness Profile Registry
The main ambition of the Openness Profile is to enable researchers to make their OS activities visible as an independent information entity, leading to a more diverse consideration of Open Science in research and related assessment. The diversity of Open Science contributions is catered for through a flexible ingestion of different types of entries, including quantitative and qualitative information. Qualitative information is captured through narratives, which facilitates structured and evidence-based input in supporting research assessment. The Openness Profile narrative can give meaning to the collection of contributions listed in the portfolio.
The latest iteration of the Open Science Assessment Framework represents a significant step forward in promoting responsible and Open Science-aware research evaluation practices. By combining comprehensive resources with innovative digital services, the framework supports organisations in aligning with global research assessment reform efforts.
This article was written by FRANCESCO RACITI (CNR)
