THE FUTURE OF COCHRANE EVIDENCE SYNTHESIS

The case for vital transformation in a rapidly changing world

Case presented by Karla Soares-Weiser, Cochrane's Editor in Chief, for consideration on 9th February 2022 Board meeting and adapted to include and reflect the approved option.

The proposal was developed and endorsed by Cochrane's Executive Leadership Team and senior team members of the Evidence Production & Methods Directorate.

Index

• Introduction
• Strategic case for change
  • Funding challenges
  • Goals and principles
• Approved option
  1. Cochrane thematic groups
  2. Cochrane evidence synthesis units
  3. Central editorial service
  4. Simplification of systems, processes, and tools
  • Key benefits of the approved option
  • Governance and accountability
  • Fundraising
  • Proposed timeline for the programme of work
  • Programme governance and management

Introduction

Our funding, structure and future sustainability are increasingly at stake. A new model of Cochrane evidence synthesis production was proposed to secure Cochrane's future and has been adopted by the Governing Board. The central aim of this model is to improve facilitation of the production of relevant and timely evidence syntheses that meet global health challenges, while upholding Cochrane's vision and principles in line with the Strategy for Change . Implementation of the model will take place in a stepwise manner over a three to five year period to allow for community input and for the model to be adjusted where necessary.

Cochrane has used the same production model since the organization's inception in 1993. Our organizational structure has grown organically over that time, resulting in inconsistent, time consuming and cumbersome processes, no overarching prioritization framework and poor author retention. Compounding this, our systems and tools are complex and unwieldly, and have a negative impact on our ability to innovate. Figure 1 provides a snapshot of the key problems with our current production model. The new model set out below, which has been approved by the Governing Board, seeks to address these challenges through a phased programme of change.

Figure 1: Challenges to systems and process to produce Cochrane reviews

The first version of the new model was shared with the Cochrane community in September 2021 and then discussed extensively through a community engagement process. The revised model for producing Cochrane evidence syntheses set out below will enable Cochrane to:

1. respond to important global health and social care needs;
2. streamline the way we produce high-quality evidence syntheses by simplifying editorial and systems and improving efficiency and consistency; and
3. achieve financial sustainability in an environment where infrastructure funding is diminishing, and funders increasingly require the research output they fund to be open access.

While Cochrane is still widely recognized as the "gold standard" producer of evidence syntheses, there is clearly scope for improvement. The potential for our reputation to be impacted by inconsistent quality, slow delivery and poor author experience is a real danger. A transformation of Cochrane's production model is vital to secure our future as a global leader in evidence synthesis production and standard setting.

Strategic case for change

Cochrane has been a global leader in evidence synthesis and standard setting for close to three decades. Our commitment to producing high-quality evidence remains unchanged, but the environment in which we operate has shifted fundamentally. As this business case demonstrates, real change is now essential to secure and sustain our organization in an increasingly competitive evidence synthesis marketplace.

This transformation must address the following key issues.

• Cochrane's organic growth has resulted in a labyrinthine organizational structure that is confusing to both internal and external stakeholders, and complex to operate within.
• In an increasingly competitive environment where less than 5% of evidence syntheses are published by Cochrane, we need to find new ways of demonstrating our value.
• Our editorial independence is open to challenge due to both the authoring and editorial processes being conducted within a single Cochrane Review Group.
• Researchers worldwide use Cochrane methods and tools, but many decide to publish elsewhere because our software, systems or editorial processes are too cumbersome and inaccessible. This hinders our ability to optimize skills and expertise in the community and deters would-be contributors from becoming involved with Cochrane.
• Organizational income is less secure. Cochrane needs to diversify its revenue streams.
• Cochrane has made a commitment to be fully open access by 2025. This aligns with our mission of making evidence accessible to all but creates an additional significant financial challenge and a need to focus on a more sustainable approach to evidence synthesis.
• Worldwide, the ways people access, share and use healthcare evidence has completely changed over the past three decades. Cochrane's structure, processes and review formats have not evolved in parallel to meet stakeholder needs and expectations.

Over the years many attempts have been made to address these issues, improve quality and processes, and reduce the burden on Review Groups (see Figure 2). Most were only partially successful, and some have added complexity to the current model; the creation of Networks is an example of this. The Network model was intended to improve the quality and relevance of reviews through standardizing editorial processes and prioritization, providing quality assurance, sharing good practice, and promoting greater collaboration between Review Groups. At a cost of approximately £0.8m per year (to run the Networks) these objectives were only partially achieved.

There was some improvement in the quality of high priority reviews but overall they proved a weak mechanism for improving relevance and quality generally, and usability of all reviews, because lack of accountability of Review Groups to Cochrane remained. There is no evidence that Networks helped to address efficiency of process or contributed to reducing the length of development for reviews, indeed, the median time from published protocol to complete review has been approximately three years since 2018.

Figure 2: Initiatives over time used to "fix" the process and systems challenges

Funding challenges

Funding has also become an increasingly critical issue for Cochrane and the Review Groups. As of March 2023, 21 Cochrane Review Groups based in England will no longer receive infrastructure funding because of a recent decision by the National Institute of Health Research. Similar decisions to end funding have been made by government agencies in Scotland and Northern Ireland meaning that in total, the 24 UK-based Review Groups will be affected by this complete loss of funding in 2023.

Another three Cochrane Review Groups (one each in Portugal, the Netherlands and Germany) did not receive any funding in 2020. Two of these groups stopped almost all review production activities in early 2021, to concentrate on a handover of work in progress to the Cochrane central team. We anticipate other funders may reconsider their investment if Cochrane does not adapt.

As a result, from 2023 Cochrane could lose half of our published output (published reviews and updates) unless we act now. We are developing a plan to keep a minimum number of reviews flowing through the publication pipeline to support the subscription business model for the Cochrane Library in 2023 and 2024.

Cochrane's reputation alone is no longer sufficient in the context of funding constraints and competition. National funders want assurance they will receive the evidence syntheses for which they pay, on the topics they consider important, and within an acceptable timeline. It is no longer realistic to expect core funding without that level of accountability. Furthermore, to secure new funding we will need to be able to meet similar funder requirements on quality, relevance and timeliness. The new model sets out a way to transform the way we produce evidence synthesis to maintain the quality and number of Cochrane evidence syntheses our users need.

Goals and principles

It is essential that the new production model aligns with Goal 1 of Cochrane's Strategy for Change - producing trusted and timely synthesized evidence addressing the most important questions for health and care decision making and is underpinned by the four enabling principles of:

• improved efficiency: the model must make use of our existing content and methodological expertise, streamline our processes, and technology and embrace other innovations to ensure that our content is consistently relevant, timely, and of high quality;
• increased sustainability: the new model must encourage and support investment from a more diverse range of funders;
• increased awareness and impact: for Cochrane to realize our vision of a world of better health for all people, we must champion innovations that will allow us to diversify the types of products we deliver and make them accessible to all;
• enhanced accountability: the new model must have clearer lines of accountability to ensure performance standards are met and that all those engaged within Cochrane understand their roles and responsibilities.

The first version of the new model (Figure 3) was shared with the Cochrane community in September 2021 and then discussed extensively throughout a community engagement process that ran from September to November 2021.

Figure 3: "Subway map" introducing a new model for producing Cochrane evidence synthesis
 

Approved option

For Cochrane to maintain its leadership position as a producer and standard setter for evidence synthesis we must adapt and respond proactively to the rapidly changing landscape. Following the consultation on the future of evidence synthesis proposals, the Governing Board, on 9^th Feb 2022, approved the proposed change to Cochrane's evidence synthesis production model and the immediate move to implementation activities.

The Governing Board has asked the Executive Leadership Team to:

• Work with partners to set up a small number of externally funded Evidence Synthesis Units (ESUs), to be located in both in high- and low- or middle-income countries;
• Develop collaborative arrangements across the community to ensure we maintain our valuable skills and expertise. These new groups will be shaped thematically and work in collaboration with other Cochrane entities. They will focus on global priorities and support the developmental and editorial processes for evidence syntheses, knowledge translation, and stakeholder engagement;
• Expand the Central Editorial Service to handle the editorial process for all evidence syntheses published on the Cochrane Library, including a direct pathway and a fast-track service, to strengthen consistency and delivery; and
• Undertake targeted projects to simplify Cochrane's systems, processes, and develop our tools to enhance efficiency in the production of evidence synthesis.

Given the scale of transformation proposed a stepwise approach will be adopted to allow for key decisions to be tested and adapted, and new sources of funding to emerge to support full implementation. Due to significance of the programme of work, the Governing Board has decided to set up an advisory group to ensure the management of this transition has ongoing Board oversight. As implementation plans are developed the community engagement process that began in 2021 will continue into 2022 and beyond.

Description of the new model

1. Cochrane thematic groups

Overview

To maintain topic expertise and relationships with internal and external Cochrane stakeholders and provide another potential mechanism for developing Cochrane evidence syntheses, ~20 Cochrane thematic groups will be established. The idea for these thematic groups arose from two proposals received during the engagement process.

The thematic groups will liaise with external networks, such as guideline developers, government agencies, and disease associations. They will ensure that our existing network of enthusiastic volunteers is preserved and that the Cochrane Evidence Synthesis Units and the Central Editorial Service have access to the right topic expertise. They may also develop Cochrane evidence syntheses and submit them to the Central Editorial Service for consideration. Key participants, roles and funding mechanisms are outlined in Table 1. Input from the Cochrane community will be key in determining how these thematic groups can be developed and work efficiently with other elements of the model.

To ensure a coordinated and holistic approach to topic prioritization the thematic groups, which will have good working relationships with key stakeholders, will collaborate with Fields, Geographic Groups, Evidence Synthesis Unit leaders and the Central Executive Team to ensure that Cochrane is responding to global health and care needs.

Table 1: Key participants, roles and funding mechanisms for Cochrane thematic groups

Selection process

The following criteria will be considered for any research group seeking to set up a Cochrane thematic group:

• Access to a network of experts who can support prioritization and evidence synthesis production within a specific health and care topic area;
• Established links with patient and client networks within a specific health and care topic area;
• Strong connections with stakeholders and policy makers, both local and international;
• Experience of innovative dissemination practices to complement evidence synthesis production;
• Evidence of capacity building activities;
• Experience of knowledge translation (optional activity).

Additionally, for thematic groups that opt to produce Cochrane evidence syntheses, relevant expertise will be required.

A full specification document will be developed as part of the implementation plan.

Funding

Cochrane thematic groups will not be funded centrally. Their leaders will be responsible for identifying suitable funding mechanisms for their activities; however, the Cochrane Central Executive Team may consider limited administrative and fundraising support in 2022 and 2023 while the thematic groups are establishing themselves.

2. Cochrane Evidence Synthesis Units

In addition to the Cochrane thematic groups, Cochrane will create a number of multi-topic, interdisciplinary Evidence Synthesis Units. Within the Units, a core team will be responsible for delivering relevant, high-quality, and timely evidence syntheses that responds to the needs of our diverse stakeholders.

Overview

Evidence Synthesis Units will be responsible for preparing evidence syntheses according to an agreed set of processes and standards, that will be defined in a contract with Cochrane and the Unit funder(s). The Units will be multi-topic, interdisciplinary and geographically dispersed, with scope for sharing expertise between Units. Each Evidence Synthesis Unit will be led by an experienced Cochrane evidence synthesis expert, who will have an appropriately skilled core team (minimum composition of core team is indicated in Table 2) and will be expected to produce 30 reviews and 15 protocols per year. The Units will work with thematic group leaders to ensure that the highest impact topics are prioritized for delivery.

Although the Evidence Synthesis Units will focus primarily on creating high-quality content for the Cochrane Library, they may also work with Methods Groups to become early adopters and standard setters for emerging methods.

The Unit leader will be responsible for reporting regularly to Cochrane and the funder using an agreed monitoring framework which will include publication output, evidence of impact and details of any other activities undertaken (e.g., dissemination, capacity building).
 

Selection process

The following criteria will apply to any research group seeking to set up a Cochrane Evidence Synthesis Unit:

• Expertise in a variety of evidence synthesis methods (intervention, diagnosis, prognosis, qualitative, etc.);
• Capabilities and sufficient capacity to produce evidence syntheses in a wide range of research topics, relevant across healthcare, public health, and social care within agreed time frames;
• A team which includes information science expertise, risk of bias assessment, health economics and economic modelling, epidemiology and statistics and knowledge/understanding of evidence sources;
• Strong connections with stakeholders and policy makers, both locally and internationally;
• Established links with patient and client networks;
• Ability to measure and record the impact of published reviews;
• A track record of delivering high quality, timely evidence syntheses.

A full specification document will be developed as part of the implementation plan. The exact staffing composition of an Evidence Synthesis Unit will vary based on funding, context/location.

Table 2: Personnel required (indicative only) for each individual Evidence Synthesis Unit to produce 30 reviews and 15 protocols per year

Funding

The Evidence Synthesis Units will not be funded centrally so Cochrane's ability to fundraise to support the Units will be critical. During 2021 and after the UK's National Institute of Health Research (NIHR) announcement, Cochrane executive leadership continued communicating with current national funders, including NIHR, to explore their willingness to fund an Evidence Synthesis Unit in their region.

As part of the stepwise approach, funds for at least two Evidence Synthesis Units will be sought during 2022 (one in a high-income country and another in a low- or middle-income country). Setting up these two units at the beginning of the process will enable a better understanding of how they can be fully implemented, and how systems, processes and tools can be streamlined to improve efficiency, as we move towards full scale up. A stepwise approach will also help us to determine what number of Units is manageable and provides the best coverage and impact for Cochrane.

3. Central Editorial Service

The Central Editorial Service will be responsible for all editorial processing and subsequent publication in the Cochrane Library regardless of the source of the manuscript. Clear, standardized expectations regarding rejection, quality assurance, conflict of interest and problematic studies will be established. Rejected submissions can be submitted for publication elsewhere, but without Cochrane branding.

The service will maintain a direct pathway to publication for unsolicited submissions and offer a fast-track service for high-profile reviews received directly from author teams or the thematic groups. In the short to medium term the Central Editorial Service direct pathway will also ensure that Cochrane Review Groups with stable funding are able to publish during the transition period. We recommend that the income from Cochrane Library royalties is used to fund the Central Editorial Service in the short term, and for the direct costs of the service to be added to the overall costs of running the Evidence Synthesis Units in future.

4. Simplification of systems, processes, and tools

During the community engagement period, there was broad consensus that Cochrane needs to simplify and streamline its review development and publishing processes. Work is planned on the format of reviews, standardization of methods and policies, and tools to support development of reviews, including hands-on support and improvements in technology. Shortening the review format is key to making reviews easier to write and access. This will also make Cochrane a more attractive publication option, improve the author experience, and streamline editorial processes and copy-editing. Production costs will be reduced in the longer term, and Cochrane will be better placed to diversify its evidence synthesis types and make content more accessible. The project plan for this work will include interviews with stakeholders, a survey of Cochrane Library users, and the production of guidance to accompany the launch of the new format for reviews.

With the intention of simplifying editorial processes and technology, and enhancing the author experience, we will also prioritize work to standardize methodological criteria and editorial policies and make improvements that support the streamlining of review production.

A demonstration of how the new production model could work in practice is presented in Figure 4.

Figure 4: How the new model could work in practice.
This is an example assuming collaboration between two Thematic Groups and three Evidence Synthesis Units.

Step 1: Cochrane is approached by an international guideline group that wants to develop recommendations on preventing and treating obesity and overweight in children and young adults. This leads to discussions with key funders, and new evidence syntheses are commissioned.

Step 2: Thematic groups engage with internal stakeholders to set appropriate questions.

Step 3: Questions are refined by the Unit Information Specialist(s) who checks for duplication in the Cochrane Library (scoping phase). Health consumers are involved in setting questions and defining core outcomes.

Step 4: The refined questions can be answered through different types of evidence synthesis (e.g., qualitative, prognostic, diagnostic, intervention); some using rapid methods, others standard methods.

Step 5: A series of six reviews is commissioned with a timeline for completion within 12 months. Both relevant thematic groups decide to take responsibility for one evidence synthesis and the Evidence Synthesis Units are commissioned to produce the four remaining evidence syntheses.

Step 6: Proposals from authors to develop evidence syntheses are managed centrally. The Cochrane Central Team provides methodological support and access to Cochrane Crowd to speed up the process.

Step 7: Completed evidence syntheses are submitted to the Central Editorial Service within eight months.

Step 8: A dissemination plan is created, stakeholders engaged, and the evidence syntheses are published in the Cochrane Library.

Step 9: Dissemination activities, including impact report to support access and use.

[Click on the figure to view a larger version]

Key benefits of the approved option

This new model of producing Cochrane evidence synthesis offers a range of benefits to both internal and external stake holders, as outlined below.

Cochrane Library users
Policymakers, guideline developers, researchers, clinicians, consumers and citizens

1. Co-ordinated, global priority-setting to ensure key health and social care topics are covered.
2. Greater diversity of methods, and types of evidence synthesis available.
3. Improved timeliness of evidence syntheses delivery.
4. More engagement with users and clearer understanding of their information needs.
5. Opportunity to explore new ways for evidence users to be involved in Cochrane evidence synthesis production.
6. Stronger editorial independence to increase confidence in the integrity of Cochrane evidence syntheses.

Cochrane authors

1. Simpler, clearer structure to attract and retain authors.
2. More opportunities to develop different types of evidence synthesis.
3. Direct pathway to publication of high-quality submissions on important topics.
4. Stronger editorial independence leading to a clearer, more consistent and transparent editorial process for all authors.
5. Quicker rejection of unsuitable manuscripts.

Cochrane community

1. Better integration of skills and expertise of Cochrane staff and members from a range of different Cochrane groups.
2. Improved ability to demonstrate the value of our products.
3. Financially sustainable operating model to support sustainable jobs.
4. Simpler pathways to participation for all Cochrane members.
5. Clearer lines of accountability to ensure performance standards are met.

Funders

1. Clearer understanding of funder needs and opportunities for diverse, mutually beneficial partnerships.
2. A more "customer-focused" approach to interactions with all stakeholders.
3. Co-ordinated priority-setting activity that demonstrates Cochrane's commitment to meeting global health and care evidence needs.
4. Increased confidence in our ability to deliver independent, relevant, and timely evidence syntheses.
5. Greater diversity of methods, types of evidence synthesis, and people to address a wider range of funder needs.

Governance and accountability

The case for change is strong but it is understandable that there is uncertainty and trepidation about how the proposed changes will be funded and the potential impact they will have on Cochrane's volunteer community. For this reason, a stepwise process is proposed that allows for key decisions to be tested and adapted, and new sources of funding to emerge to support full implementation. A three to five year programme of work is planned, with the first year focusing on streamlining the evidence synthesis production process, centralizing author proposal management, scaling up the separation of development and editorial processes, and continued negotiations with funders regarding creation of one or two Evidence Synthesis Units.

Cochrane will set clear expectations and accountability mechanisms based on the enabling principles (efficiency, sustainability, awareness and impact, and accountability), and will continue to provide support for the development of evidence synthesis methods and technology, as well as fundraising, dissemination, and advocacy.

Cochrane thematic groups

We envisage that the Cochrane thematic groups will be self-managed and work closely with the central team to avoid duplication of effort. Any evidence synthesis produced by them will be submitted directly to the Central Editorial Service which will take responsibility for quality assurance and the decision to publish in the Cochrane Library.

Limited central administrative and fundraising support for thematic groups will be considered for the first two years of the change cycle. Any funding received to perform Cochrane activities should have a tripartite agreement between Cochrane, the thematic group leadership and funders. This important governance mechanism will ensure that Cochrane is fully aware of the obligations being entered into in its name and can assess the risks and benefits associated with that.

Cochrane Evidence Synthesis Units

We envisage that Evidence Synthesis Units will be fully funded by current or new national or international funders. These agreements will be negotiated in partnership with local Cochrane Geographic Groups, set clear expectations, and establish the mechanism of accountability. The Cochrane Geographic Groups are broadly supportive of the change process and, as national reference centres for Cochrane activities, they have important local connections which will be vital in helping to identify and negotiate with funders.

For every Evidence Synthesis Unit, an agreed contract between Cochrane, the Unit director, the host organization and the funder(s) is proposed. The agreement will include key performance indicators and clear expectations relating to the relevance, quality, and timeliness of the evidence produced by the Unit. The Unit director will set yearly plans, present annual reports of activities, funding and expenditure to Cochrane and the funder(s), and, ultimately, be responsible for the quality of work submitted by the Unit to the Central Editorial Service. Cochrane will encourage and support collaboration between the thematic groups and the Evidence Synthesis Units on grants to support thematically focused evidence syntheses that align with key global health and social care topics.

Central Editorial Service

A key recommendation in this proposal is to scale up the Central Editorial Service. This Service has successfully used the approach of an initial pilot to understand and evaluate the challenges and opportunities of standardizing processes. The service has been crucial in the delivery of the recent relevant, high-priority and high-quality COVID reviews that used diverse methods and were published using an accelerated editorial process. Fundraising plans will take into consideration indirect expenses to support the production of evidence syntheses, such as the Central Editorial Service, training support and tools. However, given the importance of maintaining a stable output as we make this transition, Cochrane might consider funding the Central Editorial Service from sources such as Library revenue, or strategic reserves, during the first two years.

Fundraising

A key measure of success for this transformational programme will be our ability to fundraise to support the Evidence Synthesis Units and activities performed by the central team. During 2022 Cochrane executive leadership and Geographic Group leaders will continue a engage with national funders to explore their willingness to fund an Evidence Synthesis Unit locally.

As part of the stepwise approach, we aim to identify funds for at least two Evidence Synthesis Units during 2022 (one in a high-income country and another in a low- or middle-income country). Setting up these two units at the beginning of the process will enable a better understanding of how they can be fully implemented, and how systems, processes and tools can be streamlined to improve efficiency, as we move towards full scale up. A stepwise approach will also allow us to identify the number of Evidence Synthesis Units that is manageable and provides the best coverage and impact for Cochrane.

Proposed timeline for the programme of work

Figure 5: Proposed timeline for the programme of work

Programme governance and management

Arrangements to be put in place to ensure the successful delivery of the "Future of producing reviews in Cochrane" programme include the following.

Governing Board Advisory Group

To provide independent oversight of the transformation programme and input on strategic directions.

Key roles of the Advisory Group

• Liaise with key internal and external stakeholders on behalf of the Project Team
• Liaise with members of the Governing Board and Council
• Provide feedback on key challenging areas, such as possible risks and mitigation strategies, and identification of ongoing funding for successful strategies
• Provide input on monitoring and evaluation
• Organize independent and impartial reviews (project assurance) at different stages of the project lifespan

Project Team

A Project Team will be assembled from senior members of the Evidence Production & Methods and Publishing & Technology Directorates to maintain project scope and progress, and manage dependencies with other projects, systems, and processes. The team will include a senior communications advisor to ensure community engagement throughout. Key roles of the project team:

• Co-ordinate activity on key deliverables
• Provide day-to-day project management and decision making, including adapting the model as we learn from initial implementations
• Delegate activities and responsibility to central Cochrane team as appropriate
• Report project progress to, and engage the Cochrane community
• Escalate issues to the Executive Leadership Team, Governing Board and Advisory Group for advice when needed

Executive Leadership Team

Cochrane's Executive Leadership Team will oversee all the strategic projects related to the "Future of producing reviews in Cochrane" programme. This executive team will provide ultimate decision-making accountability, including contract arrangements, and maintain a Programme risk register. This will form part of regular reporting to the Governing Board.

Finally:

In approving this approach, the Governing Board also considered more detailed reports on:

• Community engagement and feedback
• Comparison of alternative options
• Project risk
• Cost analysis and affordability.