A little over three years ago, the Cochrane Steering Group gave a green light to the ambitious Project Transform as part of Cochrane’s Game Changer strategic initiative. The team proposed to harness technology and Cochrane’s worldwide network of contributors for more efficient review production (see this two minute video).
To understand the issues and opportunities in review production in more detail, the team gathered ideas from contributors within and beyond Cochrane. The results of this research have recently been published in Systematic Reviews, and you can read more about the story here.
Since then progress has been made on many fronts, and now two and a half years into this ambitious four-year project, we reflect here on where the project is now and what we can expect as the project moves towards completion.
The initial phases of the project focussed on understanding key issues and working on prototype systems to test potential solutions. We are now at a phase in the project where this initial research and development work is starting to pay off. We have systems in production that are being used in expanding pilot projects, and these are becoming increasingly embedded in core Cochrane processes.
Of course, it has not been a linear path. Technology changes quickly, and the project team has had to be nimble and take advantage of new opportunities along the way. One of the great advantages of pursuing this work within Cochrane has been the tremendous support for new ideas and new ways of doing things, even if these represent a change from initial plans.
One of the team's priorities for the next phase of the project will be working with the Central Editorial Unit to define the appropriate use of new systems and efficient and adequately supported roll-out, including guidance and training. Evaluations of the new systems will be designed to provide the right information for policy decisions regarding their use. So at present, the project has two main areas of focus: ensuring efficient and appropriate use of the systems the project has developed; and continuing to invest in the next phase of research, so that further improvements in review production can be realised. Read on for an update on these activities, with links out to more detailed news. You can also take a look at our new infographic showing what tools and services are available now, coming soon and in development.
As Project Transform has progressed, it has become very evident that the crowd and machine components of the project belong together as two sides of the same coin. With each new service developed, both components have a critical role to play in ensuring accuracy and efficiency. Over the last 18 months, since Cochrane Crowd was launched, thousands of people new to Cochrane have signed up to help tackle the very real problem of information overload. The crowd has collectively identified almost 35,000 reports of randomised trials for CENTRAL. The collective accuracy of the Crowd approach has meant that not only has CENTRAL been enriched, we have also been able to create highly effective machine classifiers to help accurately identify eligible reports.
Much of our work now focuses on refining the processes and workflows to enable Cochrane Information Specialists and others to make best use of both the crowd and the machine. Not only do we want to help make CENTRAL as comprehensive as possible, and we work very closely with the Centralised Search Service initiative on this, we want to help reviewers and researchers now and into the future.
The RCT classifier offers one tangible way to do this, and we are working closely with a group of Cochrane Information Specialists to help ensure that this new technology can be accessed and used confidently when needed.
We also want to enable review teams to access the Crowd as and when appropriate for their reviews. Our Screen for Me pilot work has demonstrated that the crowd can do this work accurately and speedily.
Our work on identifying diagnostic test accuracy (DTA) studies is rapidly progressing, again thanks to Cochrane Crowd for helping to generate some of the training sets needed to build a machine classifier. Identification of DTA studies presents more challenges than with randomised trials. This means that creating machine classifiers and Crowd tasks is a little more complex, but the potential savings in terms of human effort currently expended on identifying relevant DTAs could well be significant.
In the coming months a PICO task will be rolled out in Cochrane Crowd. For this task, abstracts will be first passed through machine classifiers that will suggest candidate PICO terms for the crowd contributors to select or reject. This is a major initiative with huge potential benefits in terms of trial discovery and surveillance, and we are working closely with Cochrane’s Linked Data team and Cochrane Review Groups to develop this service.
Crowd is also in the process of rolling out a new tool: Classmate. Here, teachers and trainers of evidence based health care can access the Cochrane Crowd tasks for their students. It’s a win-win as students work on real and needed tasks, thereby learning whilst doing, and Cochrane gets the help it needs to manage the ever increasing amount of data and research being produced.
TaskExchange is Cochrane’s online platform that connects people working in health evidence within and beyond Cochrane. Launched internally in February 2016, it was opened up to the global evidence community in August 2016 and now has more than 1,200 members from across the globe. Originally focussing on systematic review projects, the scope of TaskExchange has now broadened to include guideline development. Read more about how TaskExchange connects the systematic review and guideline communities here. Our most recent platform update has added an endorsement feature, so project helpers can be endorsed upon task completion, in turn assisting others to find a helper who is well-suited to their task. You can read more about this new feature here.
Living systematic reviews
The concept of living systematic reviews (LSRs) and living guidelines is of increasing interest to evidence producers, guideline developers, decision makers, funders and publishers around the globe as a way to dynamically connect evidence and practice. Right now, LSR pilot teams and editorial groups working with Project Transform are preparing to go live with the very first LSRs to be published in the Cochrane Library - you can read the latest news about that here. You can also read here about a recent tour by Annie Synnot, Project Transform team member, and Julian Elliott, Project Transform Co-Lead, to the US and Canada, meeting with many of the groups in North America with an interest in LSRs.
Tools and services for guideline developers
Eighteen months ago, the Transform team was delighted to receive a 3-year Partnership Project grant from the Australian National Health and Medical Research Council (NHMRC). A key part of this grant is setting up demonstration projects with the Australian guideline development community to show how Project Transform’s tools and services can help enable efficient evidence and guideline production.
Steve McDonald, Project Transform Evidence Pipeline Co-Lead, has just completed an extensive needs analysis to understand how the team can best provide the evidence services required by Australian guideline developers and will now turn his attention to working with the other component leads to provide guideline developers with access to Project Transform’s innovative tools and services.
Review search and writing tools
As part of the NHMRC Partnership Project grant, the Transform team is now working with researchers at Bond University in Australia to develop, refine and integrate review search and writing tools into Cochrane’s production systems. The team at Bond is currently working with the Cochrane Editorial Unit on language guidance for a service that will create machine-generated first draft results text for systematic review abstracts based on the SoF table and forest plots.
Project Transform at the Global Evidence Summit
The Transform team is preparing for another packed schedule of activities at the upcoming Global Evidence Summit, and we’d love to see you there. We will be actively seeking your input and feedback at various workshops, group meetings and discussions, over coffee, in corridors, cafes and wherever else the opportunity arises. So if you have questions, issues, ideas or input of any kind, make sure you join our activities in Cape Town. If you can’t make it to the Summit, we’re consulting widely and providing regular updates in newsletters and via our website.
Where to from here?
By project end (December 2018), the Project Transform team, in collaboration with partners around the globe, will have developed a set of tools and services that dynamically connects health research, evidence and practice. Whether it’s using the power of the machine and crowd for surveillance of newly published research, enabling review teams to create small review tasks for completion by the skilled TaskExchange community, or keeping trustworthy evidence and recommendations up to date as living systematic reviews and recommendations, Project Transform will deliver a suite of tools, services and production models that are compliant with Cochrane methodology, reliable and efficient.
A central aim of Project Transform has always been to save review teams time and energy by improving the way people, processes and technology come together in the production of Cochrane Reviews. Our aim is to support review authors (and guideline developers) so they can focus on what they do best – analysing and interpreting health evidence in a timely way so that informed decisions about health care and policy can be made around the globe. We are well on the way to achieving this and look forward to your continued support, input and feedback!
Melissa Murano on behalf of the Project Transform team
Project Transform is funded through a Cochrane Game Changer initiative (2015-2017) and a National Health and Medical Research Council of Australia Partnership Project grant (2016-2018 / APP1114605).