This page includes frequently asked questions (FAQs) about search peer review. It is intended for members of the Cochrane Information Specialist (IS) Search Peer Review Team, Managing Editors, Cochrane authors and information specialists. These FAQs address processes for the Cochrane IS Search Peer Review Team to conduct search peer review in Editorial Manager (EM).
Questions:
-
How do I apply to join or request to leave the IS Search Peer Review Team?
-
I’ve received an email inviting me to peer review a search. What work is involved?
-
How do I review a submission? Do I read the entire manuscript?
-
Do I need to assess the strategies for all databases/sources searched for a full review or update?
-
What if a search I’m asked to peer review has already been peer reviewed?
-
Will I be asked to review the authors’ responses or revisions to my peer review comments?
-
Are there sample peer review comments available for people new to peer reviewing search methods?
-
What are appropriate comments to make at the full review and update stages?
-
Is the process for search peer review in EM still under development? Who do I send comments to?
Question 1: Is search peer review now compulsory in Cochrane?
No, but search peer review is highly recommended by the Cochrane Handbook (Chapter 4, section 4.4.8) and by the CIS Executive (2019 White Paper), and submissions to the Central Editorial Service will be peer reviewed by a member of the Cochrane IS Search Peer Review Team who was not involved in the conduct or reporting of the search. For groups who are not publishing through the Central Editorial Service, the Information Specialist Reviewer task in EM can be assigned to the Cochrane IS Search Peer Review Team or to an individual Cochrane Information Specialist. For more details about assigning peer review in EM, please see the EM Knowledge Base
Question 2: Do I need to request search peer review before the protocol, review or update I'm working on is submitted to the Central Editorial Service team via EM?
No, it is not necessary to request search peer review prior to submission to the Central Editorial Service. While search quality assurance tasks ("search QA" - see more about this in question 12) are strongly encouraged, submissions to the Central Editorial Service will receive independent search peer review by an information specialist member of the IS Search Peer Review Team.
Question 3: What is the IS Search Peer Review Team?
Starting March 2022, post-submission search peer review across Cochrane will be provided by a dedicated team of IS peer reviewers. The team is made up of volunteer Cochrane Information Specialists who have agreed to receive requests for search peer review. The IS Search Peer Review Team will perform post-submission search peer review for any Cochrane editorial team in EM, replacing the previous arrangement of searches being peer reviewed by an IS within the same Network as the Cochrane Review Group where the article was registered.
Question 4: How do I apply to join or request to leave the IS Search Peer Review Team?
Cochrane information specialists may apply to join or request to leave the IS Search Peer Review Team at any time by sending a message to support@cochrane.org
Question 5: I’ve received an email inviting me to peer review a search. What work is involved?
As a member of the IS Search Peer Review Team or as a Cochrane IS assigned the Information Specialist Reviewer Task in EM, you will be invited by email to peer review the search methods in submissions of protocols, full reviews and updates. You will complete and submit your peer review in EM. Using EM, you will assess two main elements of the submission:
- the MECIR search conduct and reporting standards for protocols, reviews and updates
- the essential PRESS elements in the primary database strategy: translation; Boolean and proximity operators; subject headings; text word searching; spelling, syntax and line numbers; and limits and filters
Question 6: How do I review a submission? Do I read the entire manuscript?
The manuscript submission will be available to you as a PDF in EM after you agree to conduct the search peer review. To complete your review, read the search methods sections in the abstract and the main text, alongside the methods sections relevant to the population, interventions and study design, and check the search strategies in the appendices. A quick read of the background and objectives is recommended to understand the topic of the review and the criteria for identifying eligible studies. It is not necessary to read the entire manuscript.
Question 7: How do I complete my assessment of the search methods?
After reviewing the manuscript, you will answer a series of questions in EM. You will be asked to declare any conflicts of interest, then about the search methods in the submission. EM uses the same questions as the recommended peer review assessments forms for protocols, full reviews and updates that are available from the Search Peer Review pages of the Information Specialists’ Portal. You may find it helpful to draft your answers using the available forms in Word before adding your comments to EM. You’ll be asked at the top of the form to provide a recommendation to the Central Editorial Service: Accept, Minor Revision, Major Revision, or Reject.
Question 8: How do I enter my answers in EM? The options do not match the recommended peer review assessment forms.
There are three answers to all questions in the recommended peer review assessment forms for protocols, full reviews and updates: A. No revisions; B. Revision(s) suggested; and C. Revision(s) required.
But in EM, there are only two answers: Yes or No. This is because the functionality in EM limits us to two options. To account for this, you will see that the option to select an answer in EM is prefaced by the question, “Are revisions suggested or required?” Please answer Yes if you have either suggested or required revisions. Begin your comments in EM with “REQUIRED REVISION 1: [...]” or “SUGGESTED REVISION 2: [...],” as this will help the editor assess the submission based on your peer review. Please number your revision requests to make it easier for authors and editors to address in their correspondence.
Question 9: What if I’m asked to peer review a different review type than an intervention review? The recommended search peer review forms and questions in EM are specific to Cochrane intervention reviews.
At present, we do not have search peer review assessment forms for non-intervention review types (e.g., overviews of reviews, diagnostic test accuracy reviews, prognosis reviews, scoping reviews, mixed methods reviews, etc.). Until these are available, please use the available forms and amend Part A (the MECIR standards) as required. Most of the questions in Part A should be considered applicable to all review types (i.e., all review types should report the sources that they searched), but please enter “N/A” in your comments and indicate if a MECIR standard is not applicable for the review type being assessed. Add a note in your comments about how you assessed the search methods (e.g., “Search methods evaluated according to Cochrane Handbook Chapter 5: Overviews of Reviews”).
Part B of the form and the EM questions for the PRESS elements can be used to assess the primary database search strategy for all review types.
Question 10: How long should it take to peer review search methods?
A search peer review assessment for a submission should take between 30 to 90 minutes to complete. Please use the questions/forms in EM to focus your work. Avoid over-analyzing or redesigning searches. CIS peer reviewers assess that search methods standards have been met for publication; CIS peer reviewers are not responsible for bringing the search methods up to standard.
Question 11: How can I peer review the search methods for a completed review if the search has already been conducted?
Ideally, search methods will be peer reviewed at the protocol stage and issues, whether serious or minor, can be addressed before the full search is run; however, some submissions may not have been assessed at the protocol stage. Search peer review of full reviews and updates may be needed to identify issues with the conduct of the search that may have impacted retrieval of eligible studies and to ensure the search methods are reported according to the MECIR standards.
If you identify issues at the full review or update stages with the conduct of the search that are unlikely to impact retrieval of eligible studies, please identify these as “suggested revisions” so they may be addressed for future updates of the review.
If you identify issues at the full review or update stages with the conduct of the search that are likely to prevent eligible studies from being retrieved, please identify these as “required revisions” so they may be included in the assessment and recommendation to the editorial team.
Question 12: Do I need to assess all the database strategies for a protocol that reports strategies for multiple sources?
No, at the protocol stage you should only assess the PRESS elements in the “primary” database strategy (i.e., the strategy that was designed first and will be translated into the other sources). The primary database strategy is most likely be the MEDLINE or PubMed search. But, if it is unclear from the protocol which is the primary database, please select one strategy to check and include a note in the assessment so the authors/editors know which database has been assessed.
Question 13: Do I need to assess the strategies for all databases/sources searched for a full review or update?
No, at the full review or update stages you are only asked to assess the PRESS elements in the primary database strategy. However, if you identify errors or issues with the searches conducted in the other sources, please share these in your assessment so they can be revised for future updates. Please also note in the assessment which strategy you checked if it is unclear which is the primary database.
Question 14: What if a search I’m asked to peer review has already been peer reviewed?
It is possible the search methods for a protocol, review or update will have been evaluated prior to submission to a Cochrane editorial team. These pre-submission activities are also known as search quality assurance, or "search QA." Another member of the author team could have checked the search methods, or another information specialist may have reviewed the strategies. Search QA is highly encouraged to improve the quality of searches but is considered a separate activity to search peer review. Search peer review is an independent process conducted after manuscript submission to assess the mandatory MECIR standards and essential PRESS elements. To assess Cochrane search methods to a consistent standard, you may be asked to peer review submissions that also received pre-submission search QA.
Question 15: What form of acknowledgement do members of the IS Search Peer Review Team receive for peer reviewing submissions in EM?
If the submission that you peer reviewed is accepted and you provide permission, an acknowledgement will be published in the protocol, review or update. You will also gain Membership points for completing the peer review. Points will appear next to the peer review contribution in your Cochrane Account after all required reviews are completed for the submission. The names of peer reviewers from the last calendar year will also appear on Cochrane Review Groups' websites, unless the peer reviewer has not consented to this.
Question 16: Will the search peer reviewer's comments be sent to the individual responsible for conducting the search, even if that person is not an author on the submission?
Your assessment of the search methods will be compiled with comments from other peer reviewers, and this feedback will be shared with the authors. If there are suggested or requested revisions to the search methods and the individual who contributed the search methods is not an author, it will be the authors’ responsibility to share these revisions with the appropriate team member to address.
Question 17: Will I be asked to review the authors’ responses or revisions to my peer review comments?
The Managing Editor will assess the author revisions and, if they are unsure whether the authors have sufficiently addressed all comments, they may request that you re-review the revised version.
Question 18: Are there sample peer review comments available for people new to peer reviewing search methods?
Please see sample peer review assessment forms available from the Search Peer Review pages of the Information Specialists’ Portal.
Question 19: What are appropriate comments to make at the full review and update stages?
At the full review stage, comment on the mandatory MECIR reporting standards R33-R38. At the update stage, comment on the mandatory MECIR reporting standard UR3. For both full reviews and updates, you should also comment on the PRESS elements (listed above in question 5) in the primary database strategy. Please also see question 11 above for guidance on making suggested vs required revisions at the full review or update stages.
Question 20: Will there be training on search peer review?
Training is available to members of the IS Search Peer Review Team. Guidance will also be available from the Search Peer Review pages of the Information Specialists’ Portal.
Question 21: Is the process for search peer review in EM still under development? Who do I send comments to?
An end-to-end process for handling peer review in EM has been developed, including for search peer review. Some elements of the peer review process are hardcoded in EM, whereas other elements have been configured, such as email templates and forms. The email templates for peer review have been written considering Cochrane's policies and guidance, and the search peer review forms are electronic versions of the Cochrane recommended peer review assessment forms. If you have feedback on the process of search peer review in EM, please provide your feedback using this online form.
Question 22: Who can help me with search peer review questions in EM?
Once you agree to peer review the search methods for a submission in EM, you will receive an email from a Managing Editor. Please address specific questions about the submission to the editor who is managing the submission.
If you encounter any issues accessing EM or have any login questions, please contact support@cochrane.org
Last update: July 2022