Synthesizing the results of included studies
Cochrane Interactive Learning (CIL): module 6 - analysing the data
| Standard | Rationale and elaboration | Resources | |
| C61 | Combining different scales | Mandatory | |
| If studies are combined with different scales, ensure that higher scores for continuous outcomes all have the same meaning for any particular outcome; explain the direction of interpretation; and report when directions are reversed. | Sometimes scales have higher scores that reflect a ‘better’ outcome and sometimes lower scores reflect ‘better’ outcome. Meaningless (and misleading) results arise when effect estimates with opposite clinical meanings are combined. |
See Handbook Section 6.5.1 |
|
| C62 | Ensuring meta-analyses are meaningful | Mandatory | |
| Undertake (or display) a meta-analysis only if participants, interventions, comparisons and outcomes are judged to be sufficiently similar to ensure an answer that is clinically meaningful. | Meta-analyses of very diverse studies can be misleading, for example where studies use different forms of control. Clinical diversity does not indicate necessarily that a meta-analysis should not be performed. However, authors must be clear about the underlying question that all studies are addressing. |
See Handbook Section 10.10.1
|
|
| C63 | Assessing statistical heterogeneity | Mandatory | |
| Assess the presence and extent of between-study variation when undertaking a meta-analysis. | The presence of heterogeneity affects the extent to which generalizable conclusions can be formed. It is important to identify heterogeneity in case there is sufficient information to explain it and offer new insights. Authors should recognize that there is much uncertainty in measures such as I2 and Tau2 when there are few studies. Thus, use of simple thresholds to diagnose heterogeneity should be avoided. |
See Handbook Section 10.10.2 Cochrane Training resource: exploring heterogeneity |
|
| C64 | Addressing missing outcome data | Highly desirable | |
| Consider the implications of missing outcome data from individual participants (due to losses to follow-up or exclusions from analysis). | Incomplete outcome data can introduce bias. In most circumstances, authors should follow the principles of intention-to-treat analyses as far as possible (this may not be appropriate for adverse effects or if trying to demonstrate equivalence). Risk of bias due to incomplete outcome data is addressed in the Cochrane 'risk- of-bias' tool. However, statistical analyses and careful interpretation of results are additional ways in which the issue can be addressed by review authors. Imputation methods can be considered (accompanied by, or in the form of, sensitivity analyses). |
See Handbook Section 10.12.1 Cochrane Training resources: assessing RoB included studies and RoB 2.0 webinar |
|
| C65 | Addressing skewed data | Highly desirable | |
| Consider the possibility and implications of skewed data when analysing continuous outcomes. | Skewed data are sometimes not summarized usefully by means and standard deviations. While statistical methods are approximately valid for large sample sizes, skewed outcome data can lead to misleading results when studies are small. |
See Handbook Section 10.5.3 Cochrane Training resource: analysing continuous outcomes |
|
| C66 | Addressing studies with more than two groups | Mandatory | |
| If multi-arm studies are included, analyse multiple intervention groups in an appropriate way that avoids arbitrary omission of relevant groups and double-counting of participants. | Excluding relevant groups decreases precision and double-counting increases precision spuriously; both are inappropriate and unnecessary. Alternative strategies include combining intervention groups, separating comparisons into different forest plots and using network meta-analysis. |
See Handbook Section 6.2.9 and Chapter 11. Cochrane Training resource: analysing non-standard data & study designs |
|
| C67 | Comparing subgroups | Mandatory | |
| If subgroup analyses are to be compared, and there are judged to be sufficient studies to do this meaningfully, use a formal statistical test to compare them. | Concluding that there is a difference in effect in different subgroups on the basis of differences in the level of statistical significance within subgroups can be very misleading. |
See Handbook Section 10.11.3.1 Cochrane Training resources: exploring heterogeneity and common interpretation errors |
|
| C68 | Interpreting subgroup analyses | Mandatory | |
| If subgroup analyses are conducted, follow the subgroup analysis plan specified in the protocol without undue emphasis on particular findings. | Selective reporting, or over-interpretation, of particular subgroups or particular subgroup analyses should be avoided. This is a problem especially when multiple subgroup analyses are performed. This does not preclude the use of sensible and honest post hoc subgroup analyses. |
See Handbook Section 10.11.5.2 Cochrane Training resources: exploring heterogeneity and common interpretation errors
|
|
| C69 | Considering statistical heterogeneity when interpreting the results | Mandatory | |
|
Take into account any statistical heterogeneity when interpreting the results, particularly when there is variation in the direction of effect. |
The presence of heterogeneity affects the extent to which generalizable conclusions can be formed. If a fixed-effect analysis is used, the confidence intervals ignore the extent of heterogeneity. If a random-effects analysis is used, the result pertains to the mean effect across studies. In both cases, the implications of notable heterogeneity should be addressed. It may be possible to understand the reasons for the heterogeneity if there are sufficient studies. |
See Handbook Section 10.10.3 Cochrane Training resource: exploring heterogeneity |
|
| C70 | Addressing non-standard designs | Mandatory | |
| Consider the impact on the analysis of clustering, matching or other non-standard design features of the included studies | Cluster-randomized trials, cross-over trials, studies involving measurements on multiple body parts, and other designs need to be addressed specifically, since a naive analysis might underestimate or overestimate the precision of the study. Failure to account for clustering is likely to overestimate the precision of the study,that is, to give it confidence intervals that are too narrow and a weight that is too large. Failure to account for correlation is likely to underestimate the precision of the study, that is, to give it confidence intervals that are too wide and a weight that is too small. |
See Handbook Section 6.2.1 Cochrane Training resource: non-standard study designs |
|
| C71 | Sensitivity analysis | Highly desirable | |
| Use sensitivity analyses to assess the robustness of results, such as the impact of notable assumptions, imputed data, borderline decisions and studies at high risk of bias. | It is important to be aware when results are robust, since the strength of the conclusion may be strengthened or weakened. |
See Handbook Section 10.14 Cochrane Training resource: exploring heterogeneity |
|
| C72 | Interpreting results | Mandatory | |
| Focus interpretation of results on estimates of effect and their confidence intervals, avoiding use of a distinction between “statistically significant” and “statistically non-significant". |
Authors commonly mistake a lack of evidence of effect as evidence of a lack of effect. |
See Handbook Section 15.3.1 Cochrane Training resource: common interpretation errors |
|
| C73 | Investigating risk of bias due to missing results | Highly desirable | |
| Consider the potential impact of non-reporting biases on the results of the review or the meta-analyses it contains. | There is overwhelming evidence of non-reporting biases of various types. These can be addressed at various points in the review. A thorough search, and attempts to obtain unpublished results, might minimize the risk. Analyses of the results of included studies, for example using funnel plots, can sometimes help determine the possible extent of the problem, as can attempts to identify study protocols, which should be a routine feature of Cochrane Reviews. |
See Handbook Section 13.4 Cochrane Training resources: small study effects & reporting biases CIL: module 7 - interpreting the findings
|