FAQs for Information Specialist Search Peer Reviewers

This page includes frequently asked questions (FAQs) for Cochrane Information Specialists (CISs) conducting search peer review for Cochrane Review Groups in Editorial Manager (EM) and for the pilot Editorial Independence and Efficiency Project (EIEP). 

Questions:

  1. Is search peer review now compulsory in Cochrane?

  2. I’ve received an email inviting me to peer review a search for the Central Editorial Service team. What work is involved?

  3. How do I review a submission? Do I read the entire manuscript?

  4. How do I complete my assessment of the search methods?

  5. How do I enter my answers in EM? The options do not match the recommended peer review assessment forms.

  6. What if I’m asked to peer review a different review type than an intervention review? The recommended search peer review forms and questions in EM are specific to Cochrane intervention reviews. 

  7. How long should it take to peer review search methods?

  8. How can I peer review the search methods for a completed review if the search has already been conducted?

  9. Do I need to assess all the database strategies for a protocol that reports strategies for multiple sources?

  10. Do I need to assess the strategies for all databases/sources searched for a full review or update?

  11. What if a search I’m asked to peer review has already been peer reviewed? 

  12. What form of acknowledgement do CISs receive for peer reviewing submissions to the Central Editorial Service?

  13. Will the search peer reviewer's comments be sent to the individual responsible for conducting the search, even if that person is not an author on the submission?

  14. Will I be asked to review the authors’ responses or revisions to my peer review comments? 

  15. Are there sample peer review comments available for people new to peer reviewing search methods?

  16. What would be appropriate comments to make at the full review and update stages?

  17. Will there be training on search peer review?

  18. Is the process for search peer review in EM still under development? Who do I send comments to?

  19. Who can help me with search peer review questions in EM?

Question 1: Is search peer review now compulsory in Cochrane?

No, but every protocol, review and update that comes to the Central Editorial Service from review groups involved in the EIEP will be peer reviewed by an information specialist who was not involved in the conduct or reporting of the search. For groups who are not involved in the EIEP, search peer review is highly recommended by the Cochrane Handbook (Chapter 4, section 4.4.8) and by the CIS Executive for all Networks (2019 White Paper).

Question 2: I’ve received an email inviting me to peer review a search for the Central Editorial Service team. What work is involved? 

The Central Editorial Service is requesting CISs to peer review the search methods in submissions of protocols, full reviews and updates. You will complete and submit your peer review in Editorial Manager (EM). Using EM, you will assess two main elements of the submission: 

  1. the MECIR search conduct and reporting standards for protocols, reviews and updates
  2. the essential PRESS elements in the primary database strategy: translation; Boolean and proximity operators; subject headings; text word searching; spelling, syntax and line numbers; and limits and filters

Question 3: How do I review a submission? Do I read the entire manuscript? 

The manuscript submission will be available to you as a PDF in EM after you agree to conduct the search peer review. To complete your review, read the search methods sections in the abstract and the main text, alongside the methods sections relevant to the population, interventions and study design, and check the search strategies in the appendices. A quick read of the background and objectives is recommended to understand the topic of the review and the criteria for identifying eligible studies. It is not necessary to read the entire manuscript. 

Question 4: How do I complete my assessment of the search methods? 

After reviewing the manuscript, you will answer a series of questions in EM. You will be asked to declare any conflicts of interest, then about the search methods in the submission. EM uses the same questions as the recommended peer review assessments forms for protocols, full reviews and updates that are available from the Search Peer Review pages of the Information Specialists’ Portal. You may find it helpful to draft your answers using the available forms in Word before adding your comments to EM. You’ll be asked at the top of the form to provide a recommendation to the Central Editorial Service: Accept, Minor Revision, Major Revision, or Reject. 

Question 5: How do I enter my answers in EM? The options do not match the recommended peer review assessment forms.

There are three answers to all questions in the recommended peer review assessment forms for protocols, full reviews and updates: A. No revisions; B. Revision(s) suggested; and C. Revision(s) required.

But in EM, there are only two answers: Yes or No. This is because the functionality in EM limits us to two options. To account for this, you will see that the option to select an answer in EM is prefaced by the question, “Are revisions suggested or required?” Please answer Yes if you have either suggested or required revisions. Begin your comments in EM with “REQUIRED REVISION: [...]” or “SUGGESTED REVISION: [...],” as this will help the editor assess the submission based on your peer review.  

Question 6: What if I’m asked to peer review a different review type than an intervention review? The recommended search peer review forms and questions in EM are specific to Cochrane intervention reviews. 

At present, we do not have search peer review assessment forms for non-intervention review types (e.g., overviews of reviews, diagnostic test accuracy reviews, prognosis reviews, scoping reviews, mixed methods reviews, etc.). Until these are available, please use the available forms and amend Part A (the MECIR standards) as required. Most of the questions in Part A should be considered applicable to all review types (i.e., all review types should report the sources that they searched), but please enter “N/A” in your comments and indicate if a MECIR standard is not applicable for the review type being assessed. Add a note in your comments about how you assessed the search methods (e.g., “Search methods evaluated according to Cochrane Handbook Chapter 5: Overviews of Reviews”). 

Part B of the form and the EM questions for the PRESS elements can be used to assess the primary database search strategy for all review types. 

Question 7: How long should it take to peer review search methods?

A search peer review assessment for a submission should take between 30 to 90 minutes to complete. Please use the questions/forms in EM to focus your work. Avoid over-analyzing or redesigning searches. CIS peer reviewers assess that search methods standards have been met for publication; CIS peer reviewers are not responsible for bringing the search methods up to standard. 

Question 8: How can I peer review the search methods for a completed review if the search has already been conducted?

For the EIEP pilot, search peer review will be conducted at the protocol, full review and update stages. Ideally, search methods will be peer reviewed at the protocol stage and issues, whether serious or minor, can be addressed before the full search is run; however, some submissions coming into the pilot will not have been assessed at the protocol stage and there are different reporting standards for full reviews and updates. Search peer review of full reviews and updates is needed to identify issues with the conduct of the search that may have impacted retrieval of eligible studies and to ensure the search methods are reported according to the MECIR standards. 

If you identify issues at the full review or update stages with the conduct of the search that are unlikely to impact retrieval of eligible studies, please identify these as “suggested revisions” so they may be addressed for future updates of the review. 

If you identify issues at the full review or update stages with the conduct of the search that are likely to prevent eligible studies from being retrieved, please identify these as “required revisions” so they may be included in the assessment and final recommendation of the Central Editorial Service.  

Question 9: Do I need to assess all the database strategies for a protocol that reports strategies for multiple sources?

No, at the protocol stage you should only assess the PRESS elements in the “primary” database strategy (i.e., the strategy that was designed first and will be translated into the other sources). The primary database strategy is most likely be the MEDLINE or PubMed search. But, if it is unclear from the protocol which is the primary database, please select one strategy to check and include a note in the assessment so the authors/editors know which database has been assessed. 

Question 10: Do I need to assess the strategies for all databases/sources searched for a full review or update?

No, at the full review or update stages you are only asked to assess the PRESS elements in the primary database strategy. However, if you identify errors or issues with the searches conducted in the other sources, please share these in your assessment so they can be revised for future updates. Please also note in the assessment which strategy you checked if it is unclear which is the primary database. 

Question 11: What if a search I’m asked to peer review has already been peer reviewed? 

It is possible the search methods for a protocol, review or update will have been peer reviewed prior to submission to the Central Editorial Service. Another member of the author team could have checked the search or another information specialist reviewed the strategies. These search peer review activities are highly encouraged to improve the quality of searches, but they may not have assessed the mandatory MECIR standards and essential PRESS elements. Because the EIEP pilot aims to assess search methods to a consistent standard, you may be asked to assess a search for which some form of peer review has already been conducted. 

Question 12: What form of acknowledgement do CISs receive for peer reviewing submissions to the Central Editorial Service?

If the submission that you peer reviewed is accepted and you provide permission, an acknowledgement will be published in the protocol, review or update. You will also gain Membership points for completing the peer review. The names of peer reviewers from the last calendar year will also appear on Cochrane Review Groups' websites, unless the peer reviewer has not consented to this.

Question 13: Will the search peer reviewer's comments be sent to the individual responsible for conducting the search, even if that person is not an author on the submission?

Your assessment of the search methods will be compiled with comments from other peer reviewers, and this feedback will be shared with the authors. If there are suggested or requested revisions to the search methods and the individual who contributed the search methods is not an author, it will be the authors’ responsibility to share these revisions with the appropriate team member to address. 

Question 14: Will I be asked to review the authors’ responses or revisions to my peer review comments? 

The Central Editorial Service Managing Editor will assess the author revisions and, if they are unsure whether the authors have sufficiently addressed all comments, they may request that you re-review the revised version. Questions from Managing Editors in the Central Editorial Service about the search may also be directed to the Central Editorial Service Information Specialist. 

Question 15: Are there sample peer review comments available for people new to peer reviewing search methods?

Please see sample peer review assessment forms available from the Search Peer Review pages of the Information Specialists’ Portal.

Question 16: What would be appropriate comments to make at the full review and update stages?

At the full review stage, comment on the mandatory MECIR reporting standards R33-R38. At the update stage, comment on the mandatory MECIR reporting standard UR3. For both full reviews and updates, you should also comment on the PRESS elements (listed above in question 2) in the primary database strategy. Please also see question 8 above for guidance on making suggested vs required revisions at the full review or update stages. 

Question 17: Will there be training on search peer review?

Yes. There will be webinars on search peer review and training on EM offered during the EIEP pilot. Training announcements will be shared via the information specialists’ email list. Guidance will also be available from the Search Peer Review pages of the Information Specialists’ Portal.

Question 18: Is the process for search peer review in EM still under development? Who do I send comments to?

An end-to-end process for handling peer review in EM has been developed, including for search peer review. Some elements of the peer review process are hardcoded in EM, whereas other elements have been configured, such as email templates and forms. The email templates for peer review have been written considering Cochrane's policies and guidance, and the search peer review forms are electronic versions of the Cochrane recommended peer review assessment forms. If you have feedback on the process of search peer review in EM, please provide your feedback using this online form.  

The EIEP has developed processes for the pilot for types of peer review and peer reviewer identification. These processes will be reviewed during the progress of the pilot but the Project Team and Working Group, and changes will be made if necessary. Comments can be shared with Rachel Marshall, Editorial Lead, Editorial & Methods Department, rmarshall@cochrane.org, or Anne Littlewood, CIS representative on the EIEP Working Group, a.littlewood@manchester.ac.uk  

Question 19: Who can help me with search peer review questions in EM?

Once you agree to peer review the search methods for a submission to the EIEP, you will receive an email from an editor in the Central Editorial Service team. Please address specific questions about the submission to the editor who is managing the submission. 

If you encounter any issues accessing EM or have any login questions, please contact support@cochrane.org 
If you have general questions about the search peer review process, please contact Rachel Marshall rmarshall@cochrane.org or Anne Littlewood a.littlewood@manchester.ac.uk.