The GraspOS pilots and services teams reconvened in Athens in May 2024 for a two-day workshop, and made significant progress in choosing appropriate methods and tools for context-specific research assessment.
The workshop’s main aim was to allow the pilots to present their progress and their critical use case scenarios, and for services to identify how to respond to these needs. These discussions were translated into the Pilots’ roadmaps that will be included in the Deliverable 5.2, which will be made available in September 2024.
Halfway into the project, the pilots’ focus is on testing and evaluating the Open Science Assessment Framework (OSAF) developed in GraspOS, with the main aim to facilitate the design and planning of more responsible evaluation. To do so, four workshops were held up to now, exploring the first stages of the SCOPE Framework for Research Evaluation which is embedded in the OSAF.
The short report below summarises the latest workshop in Athens from 29-30 May 2024, which was dedicated to the third stage of the SCOPE Framework: options for evaluating. A summary of the previous three workshops is already available here.
Workshop Highlights
The meeting in Athens brought together the Pilots and Services teams of the project to discuss the nine pilots' case studies and the adoption and exploitation of the GraspOS services, data and tools.
Building on the work carried out in previous workshops, that is, the definition of values and context, participants concentrated on choosing appropriate methods and tools for the evaluation, exploring both qualitative and quantitative options for evaluating the identified values in the chosen context. The main learnings are summarised below.
Evaluation is context-specific
While in previous workshops, pilot representatives were grouped into three pilot categories, based on their level (national, institutional, and thematic), in this last encounter, the approach was different.
Initially, it was expected that pilots grouped by evaluation level would have similar needs for tools, services, and infrastructures. However, this expectation was countered by the observation that evaluation settings are unique, they happen in a variety of contexts and bear different values, which consequently leads to different choices for evaluation options.
Each pilot representative therefore presented their unique evaluation case to the GraspOS service representatives, highlighting which tools, services, and infrastructures they found the most relevant for their specific setting, and what would be needed for their evaluation to work.
What the GraspOS pilots needed to move to the third stage of the SCOPE Framework, “options for evaluating”, was to gain a deeper understanding of how to implement and test the GraspOS resources in practice.
Each pilot represents a unique use case
By treating each pilot as a unique case, both pilots and service providers teams were able to focus on specific issues which needed to be addressed before moving on to the next phases of the SCOPE Framework, the testing and evaluation stages. Following a simple structure composed of three elements, each of the nine pilots presented:
- User stories in the form of simple statements focusing on what they wish to achieve in GraspOS,
- Pilot insights on how the tools, services, and infrastructures could help make these user stories a reality,
- Pilot insights on what is preventing the realisation of the user stories in the context of GraspOS, focusing on what is deemed to be missing from the tools, services and/or infrastructures developed in the project.
Using MIRO boards, each pilot then reported their user stories, including relevant tools and indicating challenging areas. An example is provided in Figure 2 below.
Figure 1. Example of visualisation for the Computer Science pilot.
A tailored approach to building roadmaps for the pilots
The user stories and insights were transformed into roadmaps on Day 2 of the workshop. It was crucial for the Pilots and Services teams to identify the challenges ahead and to set clear steps on the directions for each pilot regarding how they intend to implement their objectives.
Roadmaps aim to help the pilots visualise essential information in a structured approach. While every pilot is different from a context point of view, the main contents of the roadmap include:
- Steps: a numbered approach for the specific activities that a pilot should implement. Steps are not necessarily organised by priority as some of the activities can be done simultaneously. It is left to the discretion of the pilots on how to assign importance to their activities.
- Output: describing the specific output that will result from interaction with the services. This could be a monitoring service in collaboration with OpenAIRE for example, or a report produced by the pilot on the use of data or tools provided by the project.
- Tasks: a clear description of the activity that should be carried out to meet the pilot’s needs, in relation to an identified output.
- Contributors: identifying the service providers that the pilot should work with in order to properly produce the output.
- Comments: a space to write thoughts and to document progress or challenging issues for all to comment.
These five topics were used to build a standard template; however, the pilots are encouraged to adapt the template so as to fit their own understanding and to document their progress in a better way. The figure below shows how the Social Sciences and Humanities Pilot has started to build its roadmap.
Figure 2. Building the Roadmap for the Social Sciences and Humanities Pilot
The columns are slightly reorganised but the spirit and philosophy of the initial template have been maintained. The Activity column explains what has been done for the pilot so far in order to include in the roadmap the underlying work of the output. This is specific to the context of the thematic area of SSH. For this pilot, a Timeline column was added so as to ensure activities are achieved by a certain point. The choice was made to focus the pilot roadmap on what needs to be done by the end of 2024, and after a further evaluation of the achieved steps by the end of 2024, to update the roadmap for 2025. This will allow the pilot to build on existing activities and document the full steps of the process.
Overall, by giving space for each pilot to explore their needs and requirements based on their individual values and context, this fourth workshop facilitated the identification of relevant GraspOS tools, services and infrastructure for each pilot.
At the conclusion of the workshop, each pilot had sufficient knowledge on its needs and requirements to draft a context-specific roadmap to perform or suggest guidance for Responsible Research Assessment. These roadmaps will be made available in a dedicated project deliverable in September 2024.
