Anne Littlewood (Cochrane Information Specialists’ Executive), Robin Featherstone (Information Specialist, Cochrane Editorial and Methods Department)

The purpose of the Cochrane Information Specialists’ Executive (CIS Exec) is to provide advice on policy and issues related to the role of the Cochrane Information Specialist. To that end, the CIS Exec recently sent a paper on search peer review to the Network senior editors and their teams with the aim of trying to get a culture of search peer review established within Cochrane. The paper has been a collaborative effort by the Cochrane information specialist community. Although the CIS Exec were responsible for writing it, Cochrane Information Specialists gave their comments and opinions, which were incorporated into the final paper. The work that went into the paper demonstrates how collaboration and Cochrane’s Network structure can be leveraged to promote co-operation and improve the quality of Cochrane reviews.

Why peer review a search strategy?
Two papers have been published, highlighting problems in search strategies for Cochrane reviews. The first was published in 2006 by Sampson et al, the other more recently by Franco et al in 2018. These papers found that a high percentage of Cochrane search strategies had errors, in a high proportion of cases, these were issues that might potentially affect recall.

How can we mitigate these errors? Peer review seems like an obvious place to start. Cochrane protocols and reviews all go out for peer review from clinicians and methodologists. However, it is rare in many Cochrane groups for peer reviewers to comment on the search strategy. This means that in the vast majority of cases, the only person who checks the search strategy and search methods section is the information specialist responsible for writing the strategy in the first place. In the Cochrane groups where there is no information specialist – and there are some – it may be that no-one with search expertise will be checking the search strategy at all. This is where mistakes can happen. A systematic review can stand or fall on the quality of the search, and if the search is not robust and trials are missed, the review will not be robust.

Investigation of the procedures of other evidence producers suggests that peer review of search is undertaken as a routine process and that Cochrane is lagging behind. The National Institute of Health and Care Excellence (NICE) (2018) in the UK, the Institute for Quality and Efficiency in Health Care (IQWiG) (2015) in Germany, the Institute of Medicine (2011), and the Agency for Healthcare Research and Quality (AHRQ) (2014) in the US all require search peer review. The Centre for Reviews and Dissemination (2009) in the UK also recommends peer review.

The importance of search peer review was highlighted with the publication of the new Chapter 4 of the revised Cochrane Handbook for Systematic Reviews of Interventions (Lefebvre et al, 2019, section 4.4.8). The Handbook now strongly recommends that search strategies in Cochrane reviews are subject to peer review by a suitably qualified and experienced medical / healthcare librarian or information specialist.

Case example: peer-review reduces screening burden for overly sensitive search

One of the many benefits of search peer review can be saving time for the author teams who are screening the results. In this real-life example from Cochrane’s Editorial and Methods Department, a search was evaluated by a peer reviewer as being too narrow for the topic area (missing a key concept) and searching many more databases that was necessary to meet the MECIR standards. Unfortunately, the search was seen during the peer review stage for the completed manuscript and not at the protocol stage. Over 9000 (!) unique results from the original search had already been screened. In light of the missing concept though, the author team decided to revise their search strategy and repeat their screening. Further, they analysed the performance of the modified and original strategies to capture the included studies already identified.

 
The new search, using feedback from the peer review, retrieved only 2500 unique citations and captured all the included studies identified from the original search. The new search also found one new included study.
 
Revisions to the search were shown to improve the precision of the search and to reduce the screening load significantly for future updates. Sadly, if this search had been reviewed at the protocol stage it would have saved the author team upwards of 65 hours of screening time!

The MOSS search peer review pilot

The information specialists in the MOSS Network decided to run a pilot project to see if they could make search peer review work within their Network. It was informal; the information specialists organised the process via email. The search methods were peer reviewed using the Methodological Expectations of Cochrane Intervention Review standards. An existing resource, a structured tool called PRESS, was used as an aid for checking the search strategy.

Information specialists took part on a voluntary basis. Only protocols were included so that any issues with having to revise a search when the review was near to completion were avoided. Feedback was provided informally via email, and protocols went for search peer review at the same time as full peer review, with a two week turnaround so that the editorial process was not held up.

The peer reviewers estimated that the process took approximately 45 minutes to an hour. It was designed to be a “light touch” process; checking that terms used were appropriate and that the Boolean logic was all correct.

The pilot concluded in September 2018, and it was agreed that it was successful by the information specialists involved. Three protocols were reviewed in the pilot phase. Minor typos were corrected, and an error in search syntax was picked up and corrected.

Lessons learned included the value of a standardised checklist (such as the PRESS tool) and the value of a search narrative (or commentary) to go alongside the search strategy, to explain why some of the decisions had been taken around the vocabulary used. The support of the development of a process for peer review of search is included in the MOSS Network’s strategic plan (Cochrane Musculoskeletal, Oral, Sensory and Skin Network, 2018), and information specialists in the MOSS Network have continued to peer review protocols for one another.

Since the pilot, two other networks (Cancer and Circulation and Breathing) have also started to explore peer review of search in their Networks.

Recommendations of the CIS Exec

The CIS Exec recommends that peer review of search strategies in protocols using the PRESS tool should be encouraged within Cochrane review groups and Networks. The information specialists within each Network should organise and support the peer review process as suits their working processes, and decide whether/when it is appropriate to peer review a search. Use of the mailing lists to ask for help with peer review is encouraged if there is no-one available to peer review at Network level.

Training and support will be provided by the Cochrane Information Specialists’ Support Team. The team have already begun to support search peer review with resources available on the Cochrane Information Specialists’ Portal. A training webinar with information specialists from NICE is in the planning stages, with more training planned for later in 2020.

We believe that search peer review has many benefits, including continuing professional development for information specialists, the opportunity to share good practice, and the improvement of the consistency and quality of search strategies in Cochrane reviews.

Resources

• Information Specialists Executive 2019 White Paper on incorporating peer review of search methods and strategies.
• Search methods and strategy peer review assessment form for Cochrane intervention protocols - Tool integrating MECIR standards for reporting a protocol and PRESS elements to assess the main database search strategy
• Peer review of search strategies webinar recording – recording of a training webinar for Cochrane Information Specialists, held in 2019.

References

• Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. 2014 AHRQ Publication No. 10(14)-EHC063-EF. Rockville, MD: Available at: www.effectivehealthcare.ahrq.gov, accessed 18 April 2019
• Centre for Reviews and Dissemination. Systematic reviews: CRD’s guidance for undertaking reviews in healthcare, 2009. Available at: https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf, accessed 18 April 2019
• Cochrane Musculoskeletal, Oral, Skin and Sensory Network. Strategic Workplan 2018-2020. 2018. Available at: https://moss.cochrane.org/about-us/network-strategic-plan, accessed 18 April 2019
• Franco JV, Garrote VL, Liquitay CM, Vietto V. Identification of problems in search strategies in Cochrane Reviews. Research Synthesis Methods. 2018 May 15, 9(3): 408-416.
• Institute for Medicine. Standards for finding and assessing individual studies. In Finding what works in healthcare, 2011. Available at: https://www.nap.edu/read/13059/chapter/1#ii , accessed 18 April 2019
• Institute for Quality and Efficiency in Health Care. General methods. 2015. Available at: https://www.iqwig.de/download/IQWiG_General_Methods_Version_%204-2.pdf, accessed 18 April 2019
• Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall  C,  Metzendorf  M-I,  Noel-Storr  A,  Rader  T,  Shokraneh  F,  Thomas  J,  Wieland  LS.  Technical  Supplement  to  Chapter  4:  Searching  for  and  selecting  studies.  In:  Higgins  JPT, Thomas J, Chandler J, Cumpston MS, Li T, Page MJ, Welch VA (eds). Cochrane Handbook for Systematic    Reviews    of    InterventionsVersion    6.    Cochrane,    2019.    Available    from:    www.training.cochrane.org/handbook.

Sampson M, McGowan J. Errors in search strategies were identified by type and frequency. Journal of Clinical Epidemiology. 2006 Oct 1;59(10):1057-e1.