Editorial and Publishing Policy Resource
Cochrane peer review policy - GUIDANCE FOR IMPLEMENTATION

Cochrane peer review policy 
Guidance for implementation

This document includes definitions, guidance and additional information to support the Cochrane peer review policy.

The term 'Cochrane Reviews' is used to refer to both Cochrane Reviews and protocols for Cochrane Reviews. Policy statements included in this document are highlighted in bold, italic text.

In this policy the term 'peer reviewer' describes someone who peer reviews a manuscript (previously referred to as a peer referee) and the term 'review author' refers to the author of a Cochrane Review.  

1    Aim of peer review 

Peer review is a term that describes the objective evaluation of clinical and scientific research, usually (but not restricted to) before publication.  All new Cochrane Reviews undergo peer review, and most updates of Cochrane Reviews also undergo peer review. Please see Section 1 of the Cochrane peer review policy for further information.

Protocols for Cochrane Reviews are peer reviewed to ensure that the research question is valid, the methods suggested are appropriate, and to avoid duplication of effort. Editors may use peer review reports to ensure that resources are allocated appropriately; for example, to prioritise protocols that answer the most relevant questions. Protocols for Cochrane Reviews may be rejected before or after peer review; for example, if the topic is not relevant or not suitable for a review, if the methodology is unsound, or if there are other major issues with the protocol.

Cochrane Reviews are peer reviewed to ensure that they follow the published Cochrane Protocol (or any deviation from the published protocol is sufficiently explained); the research question is still valid, to identify whether any relevant and important studies have been excluded, the clinical context is correct and up-to-date, the methodology is appropriate and that the conclusions are based only upon the data available. Cochrane Reviews may be rejected before or after peer review; for example, if the methodology is unsound, or if the authors are unable to revise the review to the satisfaction of the peer reviewers.  For more information, see the policy on the rejection of Cochrane Reviews.

2    Number and expertise of peer reviewers

As a minimum standard, every Cochrane Review will be peer-reviewed by at least one clinical/topic specialist (with a minimum of one external to the CRG editorial team) and one statistician/methodologist (who may, in certain circumstances, be part of the CRG editorial team).

It is expected that CRGs aim to include at least one consumer peer reviewer per Cochrane protocol and review. See Section 4 of the Cochrane peer review policy.

If the Cochrane Review deviates from standard methods or uses complex methods, it will be peer reviewed by one statistician/methodologist who may have a role in the editorial team of the CRG.  If the Cochrane Review uses standard methods, these can be checked by an appropriate member of the CRG editorial team.

Wherever possible it is highly desirable to use the same peer reviewers for the protocol of the Cochrane Review, the full Cochrane Review, and any updates of the Cochrane Review. If this is not possible, it may be helpful to send key points from previous peer review reports to the current peer reviewers to ensure that time is not wasted revisiting previous decisions (for example, on the scope of the review).

At least one subject-specialist peer reviewer must be external to the CRG editorial team (i.e. has no current editorial role). Fields are a potential source of subject peer review expertise external to the CRG editorial team.

The number of peer reviewers and the mix of expertise should be appropriate for the topic and complexity of the Cochrane Review (for example, Reviews covering a multidisciplinary topic might need to seek several specialist peer reviewers). Peer reviewers with different skills (e.g. methods, specialists) should be guided to look at specific aspects of a review (for example, it might be appropriate to guide methodologists to comment only upon the methods section of the Review).

The number and type of peer reviewer recruited to comment upon each Cochrane Review will vary, but may include content specialists, systematic review specialists, methodologists, statisticians, information specialists and consumers or other potential users of the review. 

2.1    Peer review of Cochrane Reviews of diagnostic test accuracy

The peer review process for Cochrane Reviews of diagnostic test accuracy (DTA Reviews; see section 1.3 of the Cochrane peer review policy) is managed centrally under the direction of the Editorial and Methods Department (EMD) to assure scientific quality and manage the limited peer review expertise for these reviews. Each Cochrane Review is managed through the DTA peer review process (which runs in parallel with the CRG peer review process) by a DTA contact editor who returns peer reviewer and editorial comments to the CRG for transmission to the author.

2.2    Fields

When a Cochrane Review covers a topic that is relevant to the scope of a Cochrane Field, it is advisable to contact the Field to obtain a subject-specialist peer reviewer. Cochrane Fields have access to methodologists, clinicians, policy-makers and consumers with expertise and experience relevant to their topic area. Obtaining peer review input from a Field can ensure that the review has addressed all methodological and clinical issues appropriately and has maximum relevance to, and impact upon, the external stakeholders in this area. 

2.3    Consumers as peer reviewers

Consumers have an important role to play in the peer review of a Cochrane Review, and it is an expectation that all CRGs seek involvement from consumers, or from other potential users of the Cochrane Review. Consumer peer review ensures that Cochrane Review questions are relevant to people requiring and accessing health care, and that meaningful outcomes and potential harms are considered.

In particular, the role of consumers in the peer review of protocols for Cochrane Reviews is to highlight or identify additional outcomes of importance. Consumer peer reviewers also check the language used in a Cochrane Review, ensuring that the Review is sensitive to consumers, medical terminology is used sparingly and jargon is explained wherever possible; the intention is that reviews can be read easily by a wide audience.  Note that, in some cases, consumers are also topic specialists.

For additional guidance on seeking a suitable consumer peer reviewer, please see the ACTIVE project resources available from the Cochrane Training website.

2.4    Study authors as peer reviewers

It may be necessary for authors of trials or studies included in a Cochrane Review (study authors) to be invited to be peer reviewers; for example, when the field is small and it is difficult to find peer reviewers with expertise in the field. This is acceptable as long as at least one other peer reviewer is external to the CRG editorial team and independent of any studies included in the review. The fact that a peer reviewer is an author of an included trial must be made clear in the conflict of interest statement, and the authors of the Cochrane Review should be made aware of this when receiving peer review comments.

3    Inviting peer reviewers

It is best practice to contact new potential peer reviewers at an institutional email address rather than a generic email address (for example, Gmail, Yahoo, etc.) to reduce the risk of peer reviewer fraud.  This can occur when someone other than a legitimate peer reviewer is able to complete a peer review report by intercepting an invitation email. Of course, there are circumstances when this might not be possible and, in such cases, the CRG editorial team should be aware of the risk and proceed with common sense. 

The invitation email to the peer reviewer should contain the following:

  • The title, abstract and plain language summary of the Cochrane Review (or title only if a protocol);
  • Deadline for return of comments;
  • A statement that all of the information shared is confidential;
  • A link to the Cochrane conflict of interest policy.

The follow-up email to the peer reviewer (after acceptance of the invitation) could include the following:

  • Published protocol (for reviews);
  • Title proposal/registration form (for protocols)
  • Full manuscript (note that this should only be shared with peer reviewers after the invitation to review has been accepted);    
  • Screening documents from the screening team at the Editorial and Methods Department, if available;
  • Review-specific information, such as completed data extraction forms and copies of the included/excluded studies (for reviews)
  • Peer review checklist – either the generic Cochrane version, the CRG-specific version, or the mandatory sections of the peer review checklist if no checklist is used (see section 4.1), including: 

– potential conflicts of interest declaration;
– permission to include the peer reviewers name in the collated peer review comments to the author, in the  
   acknowledgements section of the Cochrane Review and on the CRG website;
– mandatory statements

  • Details of who to contact with questions or concerns;
  • Specific peer reviewer guidance from the CRG*.
  • A link to the “Resources for peer reviewers” page of the EPPR, including information on PubLons, ORCID, MECIR standards and The Committee on Publication Ethics Guidelines for peer reviewers. (see Section 2 “Peer review conduct” for further information)*.

* Only necessary if you are inviting the peer reviewer for the first time. 

4    Peer review checklists

Cochrane peer review checklists can be used and modified as needed. It is not a requirement for CRGs to use these checklists, but please see below for the sections and statements that are mandatory and must be sent to all peer reviewers. These checklists provide a structured series of questions to guide peer reviewers through the process and provide structured peer review feedback. Guided peer review can be especially helpful to less experienced peer reviewers or for people undertaking named peer review. The following checklists are available from the Editorial Resources Committee website, and will be updated from time to time:

    •    External peer review of protocols
    •    External peer review of reviews
    •    External peer review of overviews of reviews [protocols]
    •    External peer review of overviews of reviews [review]
    •    Consumer review of reviews
    •    Consumer review of protocols
    •    Guidance on using the consumer reviewer checklist

4.1 Mandatory sections and statements that must be sent to peer reviewers

If a CRG does not use the standard Cochrane peer review checklists, the following sections/statements must be included in the materials sent to the peer reviewer, and the responses reviewed and recorded appropriately in Archie.

Permissions and anonymity

Yes

No

I am willing to be identified as the author of this peer referee feedback
(If no, please provide reasons below)

 

 

I am willing to be acknowledged in the published Cochrane Protocol / Review

 

 

I am willing to be acknowledged on the Cochrane [Review Group/DTA] website

 

 

Reasons:

 

Potential conflicts of interest

Do you have any potential conflict of interest?

 Yes (details below)

 No

You should declare and describe any present or past affiliations or other involvement in any organisation or entity with an interest in the outcome of the review that might lead to a real or perceived conflict of interest. You should report relationships that were present during the last 36 months, including, but not restricted to, financial remuneration for lectures, consultancy, travel, and whether you are an author of, or contributor to, a study that might be included in this review. You should declare potential conflicts even if you are confident that your judgement is not influenced.

Conflict of interest statement:

 

A statement that all information shared with the peer reviewer is confidential.

A statement that peer reviewer comments may be edited for clarity and consistency with Cochrane standards.

A statement that acceptance of the invitation to peer review for Cochrane is also taken as consent for peer reviewer details to be stored within the Cochrane peer reviewer database.

5    Managing named peer review

From January 2019, and consistent with Cochrane's core principles, including open and transparent communication and decision making, all CRGs will adopt a named peer review process, in which the Cochrane Review author and peer reviewer know each other’s names and affiliations during the peer review process. See section 2 of the Cochrane Peer Review Policy.

During the implementation phase of the peer review policy (until January 2019), peer reviewers may wish to opt out of named peer review and remain anonymous to the authors.  The following describes a suggested method for managing this process when using Archie Workflows.

When peer reviewers return comments to CRGs through whatever format (e.g. checklist or email with a list of comments), the CRG should check that the peer reviewer has agreed to take part in named peer review. If the peer reviewer wishes to remain anonymous, the CRG should add a note to the peer reviewer’s Properties sheet in Archie as follows:.

Ensure the reviewer has been assigned a Group role of Referee.
Create a new Administrative note with the Title ‘2018 closed peer review submitted’.
Add the name of the protocol/review to the Note text.

CRGs can then search for all peer reviewers that have submitted closed peer review by using the Advanced Search in Archie as follows:.

The People option should be selected at the top with the Match all rows (AND) option.
Role in Entity | Referee | Eyes and Vision Group | Active
Note Title | Contains | 2018 closed peer review submitted

6    Delegation of peer review responsibilities

Cochrane recognises the need for training and mentoring for early career researchers, and the delegation of peer review responsibilities to junior staff members is therefore permitted under certain circumstances.  For example, when the junior staff member is a subject expert and the experience is used as a training/mentoring exercise.  The invited peer reviewer must request permission from the CRG, or the DTA Editorial Team if appropriate, to delegate their responsibilities, and must provide the full name and contact details (including position and institutional email address) for the junior staff member so that they can be acknowledged appropriately.  The invited peer reviewer must sign-off on, and take responsibility for, the final peer review report.  Both the original peer reviewer and the delegated peer reviewer must return the “Potential conflicts of interest” and the “Permissions” statements from the peer review checklist; they may both be acknowledged in the permitted ways for their contributions to the peer review report. 

Note that the CRG and DTA Editorial Teams are under no obligation to grant permission to delegate peer review responsibilities


7    Declarations of potential conflicts of interest for peer reviewers

Peer reviewers must declare any potential conflicts of interest every time they undertake peer review of a Cochrane Review. See Section 5 of the Cochrane peer review policy for further details.

Standard Cochrane peer review checklists include a declaration of potential conflicts of interest (Figure 1). If a CRG uses a modified version of the checklist, it must include this question, without any amendments. CRGs that do not use the standard checklists, and the DTA Editorial Team, must ensure that this question is included in the materials sent to the peer reviewer. The response from the peer reviewer should be reviewed and recorded in Archie. 


Figure 1: Peer reviewer potential conflicts of interest statement

Peer reviewer conflict of interest statement


Peer reviewers must state their current affiliations(s) clearly. People with a direct financial interest in a particular intervention should not peer review a Cochrane Review of that intervention. If a peer reviewer is an author of an included/excluded trial or study (study author) this should be made clear in the conflict of Interest statement, but they can remain a peer reviewer, as long as there is no financial interest in the intervention.

It is the responsibility of the CRG or DTA Editorial Team to decide whether potential conflicting interests are sufficient to withdraw the participation of the peer reviewer and to invite an alternative peer reviewer. If the CRG or DTA Editorial Team considers the conflict of interest to be minor and agrees that it is unlikely to affect the judgement of the peer reviewer, the CRG or DTA Editorial Team must disclose the potential conflict of interest to the review authors when sharing the peer reviewer’s comments. If the CRG or DTA Editorial Team considers the conflict of interest to be major and agrees that it is possible that it will affect the judgement of the peer reviewer, the peer reviewer must be replaced. 
If necessary, the CRG or DTA Editorial Team may request advice from the Cochrane funding arbiter.

8    Acknowledgement 

As a minimum, the names of all peer reviewers who have submitted a peer review report or completed peer review checklist during the current calendar year will be published on the CRG website, unless the peer reviewer has not consented to this. Lists from previous years must be archived and publically accessible from the CRG website. See Section 6 of the Cochrane peer review policy for further details.

8.1 Acknowledgement on the CRG website

To generate a list of peer reviewers who have submitted a peer review report or completed a peer review checklist during the current calendar year, the following actions can be taken:

Prospectively, a new custom task should be inserted parallel to task D1 ‘Send comments to Contact Editor’ in the current active workflow or added to the workflow template. The new task could be named 'Add names of peer reviewers to CRG website' and the following text could be added to the Description (for that task) ‘Names of peer reviewers who agreed to be acknowledged to CRG’s website should be added’.

Retrospectively, the following advanced search could be run in Archie for a given calendar year.
The Tasks option should be selected at the top with the Match all rows (AND) option.
Group | Is | Name of CRG
Workflow Role | Is | External Referee
Task Name | Is | Review and comment on draft protocol
Status | Is | Completed
Start | After | 01/01/2018
End or Due | Before | 30/04/2018

The list generated would then need to be checked as the above search still catches the ‘Review and comment on draft review’ task where it has been skipped or completed (but the referee has failed to send feedback and the decision No Response was recorded against the Decision: Comments clear and complete?).

The above search will also capture Consumer Referees and these are easily identifiable from the list generated provided the person is given the Group role of Referee and in the current workflow, in the People’s tab, Referees, Type Consumer is selected.

Another suggestion for managing this process is to use the Notes tab for each peer reviewer that agrees to peer review and is happy to be acknowledged on the CRG website. Please ensure that the reviewer has been assigned a Group role of Referee. A new Administrative note should be created in their Properties sheet with the title ‘2018 peer reviewed and agreed acknowledgement’. The name of the protocol/review that the person peer reviewed could be added to the Note text.

The following advanced search could then be run in Archie for a given calendar year:

The People option should be selected at the top with the Match all rows (AND) option.
Role in Group | Referee | Eyes and Vision Group | Active
Note Title | Contains | 2018 peer reviewed and agreed acknowledgement
To display the list of peer reviewers on the CRG website, add a page to your website titled "Peer reviewers 2018". A suggested format for this webpage is below (the list of peer reviewers names can be a chronological list, an alphabetical list, separated by country or type of peer reviewer etc. The decision rests with the CRG):

We gratefully acknowledge the contributions of all peer reviewers. The Cochrane X Group aims to involve clinical peer and consumer reviewers in the assessment of all protocols and reviews before publication. The following people have contributed to the peer review process in 2018:

Clinical reviewers:
XXXX
XXXX
Consumer reviewers:
XXXX
XXXX

We also wish to acknowledge reviewers who have chosen to remain anonymous.

See Cochrane Wounds for an example of how the annual lists of peer reviewers can be displayed on the CRG website.

8.2 Acknowledgement in the Cochrane Review

Peer reviewers should be acknowledged in the published Cochrane Review, but permission must be sought from the peer reviewer (this will be requested in the peer review checklist). A suitable acknowledgement might be:  
“The authors [or the CRG Editorial Team, if you prefer] are grateful to the following peer reviewers for their time and comments: [Insert peer reviewer names and affiliations]”
 

8.3 Other forms of acknowledgment

Peer reviewers may, in some circumstances, receive payment from the CRG; for example, for undertaking a rapid review or when specialist/expert input is needed.  This is at the discretion of the CRG, who will fund this payment.
Certificates are appreciated by some peer reviewers, who are able to use the official recognition of peer review to count towards their continuing professional development (CPD), or continuing professional education (CPE).  CRGs may wish to generate certificates for each completed peer review report, on an annual basis, or to highlight the contributions of particularly frequent peer reviewers (e.g. the top 5 peer reviewers in a calendar year).  This is by no means mandatory. 

Two other avenues for acknowledging peer reviewer contributions that CRGs should be aware of are Publons and ORCID. In both instances the peer reviewer is responsible for registering with the relevant service and creating a profile. 

8.31    Publons 

Publons is a service that provides credits for peer review in a format that can be used in CVs and funding applications. See the Publons website for more information. 

Cochrane peer reviewers can set up an account on Publons and then forward their peer review acknowledgement emails from Archie (received upon submission of a peer review report/completed checklist) to Publons so that their peer review activity can be recorded to their profile.

Note that Publons has the facility to upload full peer review reports. As peer reviewers of Cochrane Reviews are bound to confidentiality until the Cochrane Review is published (see the Committee on Publication Ethics’ Ethical guidelines for peer reviewers), peer review reports of Cochrane Reviews that have yet to be published should not be posted to Publons. Any confidential comments to the Editor should also not be posted on Publons.

8.32    ORCID

ORCID provides a unique, persistent digital identifier to all registered researchers to support automated links between the researcher and their professional activities to ensure that work is recognized and attributed correctly. Peer review activity can be recorded in an ORCID account, either via a service such as Publons, or it may be added manually.
ORCID identifiers can be recorded in peer reviewers’ individual Archie accounts.

9    Collating peer review comments

Peer review comments should be collated before sending to the Authors. Ensure that peer reviewer names and affiliations are included for those participating in named peer review, and removed for those not participating in named peer review.  Peer reviewer comments may be edited for clarity and also if they are contrary to Cochrane standards; comments may also be merged or prioritised. 

10    Ensuring that authors address peer reviewers’ comments

Authors must provide a point-by-point response to peer review comments indicating how they have been addressed. CRGs and the DTA Editorial Team are under no obligation to undertake any further editorial steps until they are able to check the corrected version meets the requirements of the peer reviewers. See Section 7 of the Cochrane peer review policy for further information.

The CRG or DTA Editor with sign-off responsibility for the Cochrane Review is responsible for ensuring that authors have considered and incorporated any relevant and reasonable peer reviewers’ comments that they receive.

11    Feedback to peer reviewers

Providing comments and feedback to peer reviewers should improve the quality of peer review reports over time. Feedback is particularly valuable to new consumer peer reviewers and will motivate them to continue the relationship with Cochrane.   You might consider implementing a mentoring system, whereby experienced consumer peer reviewers give support to new consumer peer reviewers.

Upon request, CRGs or the DTA Editorial Team should provide peer reviewers with a copy of the authors’ responses to their peer-review comments (note that all post-publication peer review comments will receive a response). Note that with the release of Archie v4.12 (October 2016) there is a custom task in workflows to send feedback/thanks to referees and advisors.

12    Inviting peer reviewers to be authors

If a peer reviewer’s comments or recommendations result in major modifications to a Cochrane Review and the peer reviewer has relevant expertise, the Cochrane Review author team may invite the peer reviewer to become a co-author. The invitation must be made after the peer-review process is completed and with the agreement of the existing author team, and the peer reviewer must contribute sufficiently to the Cochrane Review to fulfil Cochrane authorship criteria ; otherwise they may be recognised in the acknowledgements section of the Cochrane Review. Peer reviewers are under no obligation to accept the invitation to become authors. 

NOTE:

The Cochrane peer review policy is available from the Cochrane Editorial and Publishing Policy Resource (EPPR). The websites of individual Cochrane Review Groups (CRGs) and the Diagnostic Test Accuracy (DTA) editorial team include specific peer review information relating to that particular group. For general queries relating to the Cochrane policy on peer review, please contact the Editorial and Methods Department (ceu@cochrane.org).  

 

Section info
Contact
Bryony Urquhart (burquhart@cochrane.org)
Describe change
New guidance for CRGs implementing the new Cochrane-wide peer review policy.
Change date
16 April 2018