The Cochrane Editorial Unit (CEU) has been undertaking pre-publication screening of Cochrane Reviews since 2013. In that time a team of editors from the CEU has assessed hundreds of submissions, and has not only identified areas for improvement within individual reviews, but also extracted and gathered data to help improve production practices across Cochrane Reviews. In the interests of making this information widely available as a resource for Cochrane contributors, Cochrane editors Newton Opiyo and Toby Lasserson have begun compiling a series of ‘Screening Notes’, which will publish periodically here on the Cochrane Blog.  A PDF of this blog post is available here.

Context
Since pre-publication screening of reviews began in the CEU, we have seen an increase in the number of reviews using GRADE and presenting Summary of Findings tables.¹ However, we continue to see variation in the use of GRADE throughout reviews. This could reflect differences in the planning of GRADE in review protocols. One of the things that we have recognized is the value that GRADE brings to the review process by providing a transparent framework for interpreting results. With this in mind we recently investigated the extent to which GRADE is specified in the protocol as part of the intended methods for the review. We audited two cohorts of review protocols to see if there has been any change in the way that this method is planned in Cochrane Reviews. We were also interested in identifying attributes of good practice in intended use of GRADE and SoF tables in Cochrane Reviews.

Sample & standards
We used two cohorts of protocols published in the Cochrane Library in August 2013 and August 2015. We selected these timepoints to coincide with pre-screening and two years after its introduction. We devised a checklist comprising eight items that we considered relevant for planning GRADE and SoF tables in protocols of Cochrane Reviews:

1. Reference to GRADE as a method for assessing quality of evidence
2. Description of GRADE considerations for assessing quality of evidence (e.g. risk of bias or study limitations, directness, consistency, precision of results, and publication bias)
3. Description of GRADE levels of evidence (high, moderate, low, very low)
4. Description of the methods for preparing SoF tables (e.g. use of GRADEpro software)
5. Consideration for comparisons to be covered in SoF tables
6. Consideration for outcomes to be presented in SoF tables
7. Description of the number of reviewers to be involved in GRADE assessments
8. Description of GRADE and SoF tables in an appropriate section in the protocol

A full description of the methods we used can be found in the full audit report²

Implications of findings
The headline results make for good reading. Importantly, we found a substantial improvement in the number of protocols describing the intention to use GRADE, from 35% to 91%. We found improved reporting of most of the aspects we assessed. A full description of the results can be found in the full audit report².

We also identified key areas for improving the planning of GRADE and SoF tables in protocols of Cochrane reviews. We present four protocols identified from the audit which illustrate learning points for authors and editors to use in their practices.   

Example 1: Total pancreatectomy and islet autotransplantation for chronic pancreatitis (Vallance 2015, CD011799)

“We will create a 'Summary of findings' table including the following outcomes – mortality, health-related quality of life and major complications. We will use the five GRADE considerations (study limitations, consistency of effect, imprecision, indirectness and publication bias) to assess the quality of a body of evidence as it relates to the studies that contribute data to the meta-analyses for the prespecified outcomes. We will use the methods and recommendations described in Section 8.5 and Chapter 12 of the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2011) and using GRADEpro software. We will justify all decisions to down – or upgrade the quality of studies using footnotes and make comments to aid the reader's understanding of the review where necessary. We will consider whether there is any additional outcome information that we were unable to incorporate into the meta-analyses, note this in the comments and state if it supports or contradicts the information from the meta-analyses.”

This first example provides a fair overview of the key aspects of planning GRADE and SoF tables in protocols: reference to GRADE methods, description of GRADE criteria for downgrading quality of evidence, methods for preparing SoF tables and outcomes to be included in the SoF table. We thought that the last sentence in particular showed that the authors were thinking beyond the results of their meta-analysis. It will be useful for readers to be aware of key information about the review findings that cannot be included in the analysis or that provides a more complete summary of the body of evidence. 

Example 2: Virtual reality simulation for reducing pain in children (Lambert 2013, CD010686)

“Results of the meta-analysis will be presented for the main comparisons of the review, the primary outcome child pain and the following secondary outcomes: child satisfaction with virtual reality simulation, child pain-related distress and parent anxiety, as outlined in the section on Types of outcome measures. For each assumed risk cited in the table(s), we will provide a source and rationale, and the GRADE system will be used to rank the quality of the evidence using GRADEprofiler (GRADEpro) software (Schünemann 2011). If meta-analysis is not possible, we will present results in a narrative ‘Summary of findings’ table format (drawing on Chan 2011 as an example).”

In this second example the authors mention plans to prepare SoF tables for the main comparisons of the review. This is particularly important where multiple comparisons are to be addressed. Specification of comparisons to be covered in SoF tables remains an area where improvement is needed. Thirty six protocols included in our audit planned multiple comparisons, but only seven of them mentioned the comparisons to be covered in the SoF tables. Selective reporting of results for a subset of review comparisons may introduce bias, for example, if reviewers only present comparisons that show treatments to be effective or less harmful, or for comparisons where there is most amount of data available. For broad review questions the full extent of relevant comparisons may only become apparent after studies are identified, and may vary depending on the analysis undertaken. Prioritizing comparisons at the protocol stage will help focus the review on addressing questions that users are likely to value most.

Another aspect of good practice in this example is that the authors provide information about the source of assumed risk. This provides a clear basis for translating and communicating relative effects into easily understandable absolute effects. By doing this users will find it useful to determine the applicability of the review findings to practice settings. Decision-makers reading the review will find this information helpful in contextualizing the results of the review.

The authors also use the protocol to provide a commitment to narrative summary of the results in the absence of a meta-analysis. This is crucial as omission of narratively synthesized outcomes in SoF tables represents a source of outcome reporting bias. This a recurrent issue in the reviews we screen. We will write more about narrative SoF tables in a separate Screening Note.

Example 3: Concomitant atrial fibrillation surgery for people undergoing cardiac surgery (Huffman 2015, CD011814)

“Two review authors (MDH, KNK) will independently rate the quality for each outcome. We will present a summary of the evidence in a 'Summary of findings' table, which provides key information about the best estimate of the magnitude of the effect, in relative terms and absolute differences for each relevant comparison of alternative management strategies, numbers of participants and studies addressing each important outcome, and the rating of the overall confidence in effect estimates for each outcome. (…)

In addition, we will establish an appendix 'Checklist to aid consistency and reproducibility of GRADE assessments' to help with standardisation of 'Summary of findings' tables (Meader 2014).”

In this third example the authors outline a process for rating the quality of evidence in duplicate. Given the subjective nature of downgrading decisions we think that this is likely to help embed a structured, consensual approach to rating the quality of evidence.  

Very few protocols in our audit reported on the number of raters involved in assessing the quality of evidence; only four protocols mentioned involvement of at least two reviewers. Duplicate or consensual processes for rating quality of evidence merit greater attention. Studies assessing the consistency of GRADE ratings support the need for more than one rater.^3,4 With this in mind consideration should be given to the use of checklists⁴ in GRADE assessments as planned in this protocol. These checklists should help improve transparency and consistency of GRADE assessments. They may be particularly helpful for those with limited experience using GRADE.

Example 4: Needle size for vaccination procedures in children and adolescents (Beirne 2013, CD010720)

“In the Types of outcome measures section of this protocol we have listed the outcomes (primary and secondary) in terms of perceived order of importance for decision-making and we will include in the 'Summary of findings' tables the first seven outcomes listed. However, as noted in section 11.5.6.2 of the Cochrane Handbook, the importance of an outcome "may only become known after the protocol was written or the analysis was carried out and [review authors] should take appropriate action to include these in the 'Summary of findings' table" (Schünemann 2011a). In the event that during the review process: a) we become aware of an important outcome that we have omitted to include in our protocol or b) we become aware that we have failed to accord sufficient priority to a specific outcome(s) listed in our protocol, then we will include the relevant outcome(s) in the 'Summary of findings' tables. If it is necessary to include outcomes in the 'Summary of findings' tables that were not pre-specified in our protocol, then we will clearly explain the reasons for this in our review, as recommended by Kirkham 2010.”

For this last example we thought that the authors had prepared a reasonably comprehensive overview of the necessary aspects of GRADE and Summary of Findings tables. However, we also noted that whilst the authors had clearly prioritized a certain number of outcomes, they were also mindful of the need to be transparent about changes to outcome selection once the process of data collection was underway. We recognize that anticipating changes to outcomes is not a minimum reporting requirement for all review protocols, but it demonstrates commitment to transparent reporting of changes to methods in the full review.    

Prespecifying outcomes (including methods of measurement, time-points of reporting) helps avoid bias in the choice of outcomes^5,6, and serves as a reminder to review teams to include results for the most relevant outcomes in SoF tables irrespective of the amount and quality of evidence. However, some modifications may be justified; for example, as noted in this protocol ‘the importance of an outcome may only become known after the protocol was written and analysis carried out’. This may be the case for adverse events. Changes to outcomes may also be necessitated by peer reviewers. In these situations authors should explain and appropriately document any change in the specified outcomes.

Summary points
We found a welcome improvement in the planning of key aspects of GRADE and SoF tables in protocols of Cochrane Reviews.  The key learning points from this audit are that:

• There has been an increase in the number of protocols that incorporate plans for using GRADE
• Protocols benefit from defining and prioritizing comparisons for SoF tables
• Duplicate or consensual processes for rating quality of evidence merit greater attention
• Early piloting and independent evaluation of the use of GRADE (once the analysis has been undertaken but before the review is written up) should help improve the implementation of GRADE in full reviews

References

1. Cochrane Editorial Unit. Audit of published new Cochrane Reviews of interventions: 2014 Target 1.2, available from http://editorial-unit.cochrane.org/mecir [date accessed: 10^th February 2016].
    
2. Cochrane Editorial Unit. Audit of planned methods for using GRADE and preparing Summary of Findings tables in protocols of Cochrane Reviews: Target 1.3 for 2015, available from http://editorial-unit.cochrane.org/mecir [date accessed: 24^th February 2016].
    
3. Mustafa RA, Santesso N, Brozek J, Akl EA, Walter SD, Norman G, et al: The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses. J Clin Epidemiol 2013, 66:736–742.
    
4. Meader N, King K, Llewellyn A, Norman G, Brown J, Rodgers M, Moe-Byrne T, Higgins JP, Sowden A, Stewart G. A checklist designed to aid consistency and reproducibility of GRADE assessments: development and pilot validation. Syst Rev. 2014 Jul 24;3:82.
    
5. Smith V, Clarke M, Williamson P, Gargon E. Survey of new 2007 and 2011 Cochrane reviews found 37% of prespecified outcomes not reported. J Clin Epidemiol. 2015 Mar;68(3):237-45.
    
6. Kirkham JJ, Altman DG, Williamson PR. Bias due to changes in specified outcomes during the systematic review process. PLoS One. 2010 Mar 22;5(3):e9810.