David Tovey is the Editor in Chief of the Cochrane Library, and has been working with the Cochrane Editorial Unit (CEU) and the wider Cochrane collaboration in this role since 2009. In this post, he discusses Cochrane's conflict of interest policy and recent calls for re-assessment of its application in the conduct of Cochrane Reviews.

This blog has been corrected on 04/05/16 to remove misleading content caused by a misunderstanding on the author’s part.

This blog has been updated on 28/06/17 to include additional information and correspondence.

I didn’t know Bill Silverman, so I can’t judge whether he would be “a-mouldering in his grave”. However, I recognise that James Coyne has set down a challenge to Cochrane to explain its approach to commercial and academic conflicts of interest and also to respond to criticisms made in relation to the appraisal of the much debated PACE study.

Cochrane is still fairly unusual within the journal world in that it specifies that in some cases declaration of interests is necessary but insufficient, and that there are individuals or groups of researchers who are not permitted to proceed with a given systematic review. This has been true since 2004, when Cochrane’s Steering Group ratified a commercial sponsorship policy that described circumstances where authorship as proposed within a review could not go ahead. At the time, Cochrane also introduced the post of Funding Arbiter, reporting directly to its Steering Group, to ensure that the policy was followed, and to rule on ambiguous or disputed cases. As Professor Lisa Bero says “The Cochrane policy is strict because, first, there are no journals that prohibit publication of systematic reviews funded by a company with a financial interest in the outcome of the review. Second, to my knowledge, there are no journals that require the majority of authors to be without personal conflicts of interest, prohibit the first author from having a conflict of interest, or prohibit company employees with a conflict of interest from being an author. For example, the BMJ conflict of interest policy states, ‘We are not aiming to eradicate such interests; they are almost inevitable’ and authors with conflicts of interest are not prohibited from being authors of BMJ original research, systematic reviews or meta-analysis articles. The BMJ does prohibit authors with COI from being authors of: Editorials and education articles (clinical reviews, practice articles, state of the art reviews, Minerva pictures, and Endgames), but these are different from systematic reviews.”

The commercial sponsorship policy was revised and re-ratified by the Cochrane Steering Group in 2014 following an internal consultation. It is an issue that excites strong views inside our community, as it does elsewhere. Within the 2014 policy it was made explicit that review authors could not be employed by pharmaceutical companies, device manufacturers or individuals that were seeking or holding a patent relevant to the intervention or a comparator product. Furthermore, in all cases, review author teams are required to have a majority of non-conflicted authors and the lead author should also be non-conflicted. The policy is available freely.

The policy does not otherwise exclude individuals on the basis of employment or people who have conducted sponsored trials in the area. These interests should be declared, but they do not discount the individuals in question from being able to join the review team if certain restrictions on participation are met. We recognise the need to update the Cochrane policy in respect of academic conflicts, and this process is in the pipeline – however current policy states that:

“Authors of primary studies should not extract data from their own study or studies. Instead, another author(s) or an editor(s) should extract these data, and check the interpretation against the study report and any available study registration details or protocol.

Also, the relevant authorship of the primary studies should be disclosed in Cochrane's disclosure of potential conflicts of interest form and therefore the Cochrane Review.”

We believe that our editorial processes also safeguard against reviews being unduly influenced by individuals with conflict of interest.

In the case of the CFS review, the currently published review is compliant with our policy given that the authors are employed by the Norwegian Knowledge Centre for Health Services, the Norwegian Institute of Public Health, or Oxford University. All have declared that they do not have any conflicts of interest.

In relation to the handling of the PACE study, Coyne reports both the criticisms made of the decision to judge it as being at low risk of bias in relation to selective reporting of outcomes, and also Larun’s response. We also need to remember that the appraisal of the PACE trial sits alongside that of seven other studies that the authors identified. It is undoubtedly true and fairly common for people to judge risk of bias differently; it is often a subjective judgement on the basis of the known factors of the extent to which an issue (in this case changes made to the conduct of the study or its analysis made post-protocol) may have influenced the outcomes. Systematic reviews should provide a balanced interpretation of the results that takes account of considerations likely to affect our confidence in the findings. Here we can see that the authors have explicitly acknowledged the lack of blinding as a source of performance bias, downgrading the quality of evidence for all the outcomes accordingly. Assessing the quality of evidence by necessity requires due consideration of all relevant sources of bias alongside factors such as the consistency of effect and sample size.

Coyne notes that the current review author team for the IPD review includes the trialists of the studies where individual patient data has been provided. This seems to be common practice in the world of individual patient data systematic reviews, although it is also important that all authors within a review team meet the requirements of the International Committee of Medical Journal Editors (ICMJE) in relation to authorship. In the case of the current protocol, the roles of the various authors are fully described, including a statement that two authors (Odgaard-Jensen and Brurberg) will “conduct all analyses for the review…independently of trialists providing data”. In the case of the proposed individual patient data review, Professor Paul Glasziou has confirmed to me that the trialists involved in the review fully meet the ICMJE criteria for authorship, and that this is in line with standard practice - acknowledging the important role trialists can play in support of the IPD review to agree common methods and definitions to apply across all trials. 

Another issue raised by Coyne has also been raised with me in personal correspondence: namely the perceived use of Cochrane as a rationale for withholding clinical trials data at the level of individual patients from other individuals and organisations. Cochrane is a strong supporter and founding member of the AllTrials initiative and is committed to clinical trials transparency. Cochrane does not believe that sharing data with its researchers is an appropriate rationale for withholding the data from alternative researchers. Each application must be judged independently on its merits. Cochrane has issued a public statement that details our position on access to trial data.

In a later post, Coyne uses John Ioannadis to underline his concerns about Cochrane following a much quoted interview with Retraction Watch. Ioannadis gave a presentation with very similar conclusions to a Cochrane Colloquium plenary in Vienna in 2015. He has also been reported elsewhere as suggesting that he was in danger of ruining the party, but as a witness I can say with confidence that this was far from the case. His presentation was warmly and respectfully received, as befits arguably the world’s foremost researcher in this area, by a community that already spends a substantial part of its time worrying about bias, and thinking about how to minimise it. Ioannadis may be correct that the primary purpose of evidence synthesis is to demonstrate the flaws in the available primary research. He may also be correct that in the future systematic reviews “should become primary research, where studies are designed with the explicit anticipation that they are part of an overarching planned cumulative meta-analysis.” However, in the meantime I believe that Cochrane Reviews characteristically present a more skeptical and balanced view of the evidence than is presented elsewhere, whether in primary or secondary research – something for which we receive much criticism from all the predictable quarters. I am confident that our reviews represent one of the most effective challenges to the optimism bias on view elsewhere in the literature, and in the minds of many practicing clinicians, but we will of course reflect on Ioannadis’ comments when considering future methods policy.

In summary, Cochrane recognises the importance of conflict of interest as a potential source of bias in its reviews. Since 2004, it has determined those circumstances where declaration of conflicts is insufficient, and either an individual or team should be disbarred from conducting a Cochrane Review. The implementation of this policy is complex and there are many areas where the decision on whether the review can proceed requires a thorough investigation. In relation to the CFS reviews, we are extremely aware of the heightened scrutiny that the IPD review will attract. We will therefore ensure that the author team is fully compliant with our conflict of interest policies, and ICMJE guidance on authorship, and that our editorial and peer review processes lead to the publication of a review that is robust and credible.

Cochrane is not complacent. We recognise that both we and the world we inhabit are imperfect and that there is a heavy responsibility on us to ensure that our reviews are credible if they are to be used to guide decision making. This means that we need to continue to be responsive and and open to criticism, just as the instigators of the Bill Silverman prize intended, in order “to acknowledge explicitly the value of criticism of The Cochrane Collaboration, with a view to helping to improve its work.”