The transition towards Responsible Research Assessment (RRA) is at odds with commercially controlled systems that underpin traditional evaluation methods, such as Web of Science or Scopus. While the academic community champions openness in scientific communication, it continues to assess research through vendor-controlled platforms that impose financial burdens, restrict access, and operate behind opaque processes. This reliance on closed systems not only hampers innovation but also perpetuates inequities, limiting participation from underrepresented regions and institutions Open Infrastructures (OIs) represent an alternative and sustainable path forward by enabling more inclusive, transparent and fair assessment practices.
Since its establishment in November 2023, the CoARA Working Group Open Infrastructures for Responsible Research Assessment (OI4RRA WG) has focused on facilitating the shift from proprietary to open infrastructures. The WG recently reached an important milestone with the publication of the report “Open Infrastructures for Responsible Research Assessment: Principles and Framework”, released on 10 February 2025. The document makes a strong case for the transition to OIs, highlighting their strengths and providing guidance and actionable principles to research institutions, funders, policymakers, and infrastructure providers. The vision of OIs that emerges is closely aligned with the Open and Federated Research Assessment Infrastructure which is being developed in GraspOS.
The OI4RRA report stresses that transitioning to OIs requires institutions and stakeholders to identify the advantages that OIs offer in comparison with closed systems. These can be summarised in four key contributions.
- In contrast to the traditional focus on publications and journal-based metrics, OIs support the consideration of a broad range of scholarly contributions in research evaluations.
- OIs have the ability to integrate data-driven indicators with the nuance of contextual and narrative based information.
- Thirdly, interoperability paired with community-driven governance for evaluations promote the uptake of best practices and foster trust.
- Lastly, an emphasis on transparent data automation streamlines workflows, allowing researchers to devote more time to actual research.
Principles for Open Infrastructures
The report builds upon the work done by several international initiatives - including the Barcelona Declaration on Open Research Information, the UNESCO Recommendation on Open Science, the Principles of Open Scholarly Infrastructure, and the EOSC Strategic Research and Innovation Agenda -, outlining nine dimensions which provide a framework to evaluate and advance OIs.
1. Community Governance: Stakeholder-driven models reflecting scholarly diversity, ensuring inclusive, representative, and responsible decision-making.
2. Transparency: Clear communication of policies, governance, and operations, alongside open access and re-use of scholarly outputs to build trust and accountability.
3. Openness: Access to data, metadata, operational processes, and software. Accompanied by open standards, protocols, and open-source software, ensuring interoperability and reusability under open licenses.
4. Sustainability: Long-term financial models with diverse funding, mitigating risks, ensuring independence and sovereignty from commercial entities, and maintaining equitable participation and benefits.
5. Diversity: Diverse academic, indigenous, and local communities, fostering inclusion and participation across regions and social actors.
6. FAIR Data: Research data adheres to the FAIR principles.
7. Equity: Universal accessibility and usability of data and scholarly outputs, adhering to metadata standards and advocating equitable participation.
8. Responsibility, Integrity, and Accountability: Safeguarding the authenticity and reliability of scholarly outputs while ensuring accessible and trustworthy infrastructures.
9. Innovation: Reflection and adaptation to technological and social changes, aligning with community values to enhance scholarly communication.
A framework to build Open Infrastructures fit for Responsible Research Assessment
The report provides a detailed outline of the key characteristics OIs should possess in order to effectively enable RRA, not only replacing closed systems but surpassing them in “robustness, resilience, and adaptability”. These are grouped into four categories, which are summarised in the image below.
Image: A visual summary of the four key categories that define OI4RRA
Technical Robustness
Quality and reliability of data and algorithms play a pivotal role in ensuring credible, transparent, and equitable research assessments. Tracking systems that document the entire life-cycle of data are equally central, as is the ability of OIs to be interoperable with other systems and platforms through the use of recognised standards (such as Persistent Identifiers and well-documented APIs).
Operational Capacity
Open infrastructures for research assessment should be user-friendly, efficient and aligned with stakeholder needs. They should accomodate a diverse range of contributions to science - such as datasets, software, and community engagement activities - while providing flexible systems that can be customised based on specific assessment contexts. For instance, by allowing institutions to modify dashboards or select indicators best suited to their goals. Operational capacity also means providing users with proper support and training activities.
Community-Centered Practices
The long-term success of OIs relies on empowering stakeholders, which should be involved in the creation, governance and oversight - through advisory committees and feedback mechanisms - of the infrastructure. The collaborative nature of OIs fosters a sense of shared responsibility and ensures that such infrastructures address the needs of a diverse range of communities and disciplinary contexts.
Ethical and Inclusive Practices
An ethically grounded evaluation system is a cornerstone of OIs. Methodologies and metrics should be well documented, transparent, and reproducible. Humans should always have oversight over algorithms, in order to avoid black-box effects and biased processes. Responsible handling of data is also to be emphasised.
What is next?
The activities of the OI4ARRA WG will focus on producing complementary outputs including a report based on interviews exploring the transition to OI, and case studies documenting transition experiences, a toolkit for assessing the suitability of OIs and a schema to describe all key characteristics of an OI for responsible research assessment, developed in cooperation with GraspOS. The latter will be translated into the GraspOS Assessment registry to help stakeholders discover, evaluate, and engage with ΟΙs that best suit their research assessment needs.
The OI4RRA WG invites you to evaluate the Open Infrastructures for Responsible Research Assessment: Principles and Framework consultation document (available here https://zenodo.org/records/14844582). To provide your feedback, complete the form (available until 14 March 2025).
As part of the consultation process, the OI4ARRA WG will also host a webinar on 27 February at 15:00 CET, providing an opportunity to engage with the team behind the framework.
