Cochrane Exercise and ME/CFS Review Update: May 2021

Report by Hilda Bastian, Independent Advisory Group Lead

Posted online: 31 May 2021

Note: Background information on this project is here. The views expressed in this report are those of the author, and are not necessarily shared by the Independent Advisory Group, the review team, or Cochrane.

Summary

Like many projects planned before the pandemic, this one was deeply affected by the upheavals of 2020. As a result, it might not be possible to reach the original goal of a finalised review in early 2022. However, that is still the aim.

Progress in four areas is reported in some detail in the body of this report:

  • gathering and analysing community concerns about the review as a starting point for the update, and as a framework for engaging with stakeholders;
  • establishing new editorial arrangements for the review;
  • recruitment of the review team, including consumer authors; and
  • formation of the Independent Advisory Group (IAG).

Some aspects of each of these processes aren't typical for updates of Cochrane reviews and we expect to learn as we move through them. The first, an analysis of community concerns, is an ongoing project. It allowed the reviewers' work to start by engaging systematically with concerns expressed about the current review, and people's suggestions for improving it. One of the working products of the process will be a paper from the IAG that will provide a framework for consultation in the coming months.

Another example that's not typical for Cochrane will be at the protocol stage. The protocol is the roadmap of methods that the reviewers will follow to find, select, analyse, and interpret the evidence. It is published after peer review. New Cochrane reviews are preceded by a protocol. However, an update is not usually preceded by a new protocol, although it does happen occasionally.

For this updated review, there will be a new protocol. After peer review, it will be released for a very short consultation period – we hope to reach that point in the third quarter of the year. There will be opportunity for stakeholders to raise their concerns in virtual meetings, as well as in writing, before the protocol is finalised and published.

The editorial management of the review will also be new. An important issue of community concern with this review was its original editorial group within Cochrane, and the review has been re-located.

The members of the review author team and the IAG provide substantial deep expertise in ME/CFS, as well as systematic review experience – a balance that aims to optimise the review's quality, relevance, and credibility to the range of stakeholders with an interest in a review on this topic. Since Cochrane posted in March 2020 that we were considering authors and establishing the IAG, expressions of interest and suggestions about potential contributors have been coming in. We are very grateful to everyone for their interest and willingness to contribute. Everyone has been considered for each of the roles, and will be considered for the range of other activities currently planned for the review – for example, peer review and a discussion group of ME/CFS groups.

Although work has been progressing, this report and wider stakeholder engagement in the process is very long overdue, and I apologise for that. We will report on the project's status and progress at least monthly from now on.

The IAG will be responsible for responding to stakeholders and other interested parties, so enquiries and feedback relevant to this review will be referred to us. You can contact us directly at cochrane.IAG@gmail.com

I look forward to the discussions to come.

Hilda Bastian
28 May 2021

Contents:

  1. Content analysis of critiques of the Cochrane review 
  2. Editorial arrangements for the review
  3. Review authors
  4. Independent Advisory Group (IAG)

1. Content analysis of critiques of the Cochrane review

The re-consideration of this review is due to the commitment many people made to articulating and publishing their concerns about its contents and conclusions – as well as pushing for change and inclusion of people with ME/CFS. In particular, Robert Courtney and Tom Kindlon submitted influential major detailed comments that Cochrane published, but did not adequately respond to.

So the starting point for the update of the review was the need to deeply engage with “historical" stakeholder critiques. This part of the process provides a framework to ensure that issues are thoroughly addressed from the outset, to assist in consultations and incorporating ongoing feedback, and to systematically address the points raised.

Step 1: Identifying critiques

I identified 28 published critiques by internet searching. I compiled these into a public Google spreadsheet, and in May 2020, circulated this to the early members of the IAG, engaged in discussion in the Science for ME online forum, and posted it on Twitter to ensure the collection of comments was as comprehensive as possible, particularly for those made by people with ME/CFS.

After a suggestion by MSEsperanza on the forum, an additional open sheet was set up, to which material could be added by anyone. These comments did not need to meet the criteria of directly addressing the Cochrane review, and being publicly available. Later, several unpublished documents submitted to the IAG were added, including two papers on related methodological issues submitted by Science for ME. Some discussion threads on Twitter and the Science for ME forum were also included.

This information gathering expanded the collection of critiques to 48 sources, from 2008 to 2020. (The protocol for the original review was published in 2001. The first version of the review was published in 2004, and the current version was published in 2019.) Many thanks to everyone who helped fill out this collection!

Step 2: Method for content analysis

I developed a system for distilling unique points into summaries directly related to the systematic review, with coding to map them to related parts of the review. The method included checking the 2019 version of the review to identify whether items still applied. (A code for “no longer relevant" was an option.) The coding process was refined after feedback from the early members of the IAG.

Step 3: Content analysis

I did the initial content analysis, distillation of discussion points, and coding. Alongside this, I added literature referred to within discussion points, or otherwise potentially relevant or critical reading, to a reference management system collection (Zotero).

The same or similar discussion points arose frequently. A point was only added once, sourced to the first critique that raised it. There were over 100 different discussion points, although with some overlap, and some were coded “no longer relevant". There were points relevant to all sections of the review except the search strategy and data management.

These three resources – the spreadsheet of critiques, the list of literature, and the first rough distillation and mapping of discussion points – are being considered by the review authors and the IAG.

Next steps:

The points will be refined in discussion with the IAG. We expect to develop a document for consultation with the public, and the authors of comments/critiques, based on the material. We will also expand and refine the list of points as engagement with the community identifies additional issues that need to be considered and addressed.

The IAG will also consider the issues raised that are not specifically related to the content of the review itself, including issues about Cochrane's processes and how the review was handled.

2. Editorial arrangements for the review

The previous editorial arrangements for this review had been problematic for several reasons. For example, not including consumers at any level in the editorial process, and having non-psychological interventions for ME/CFS under the umbrella of the Common Mental Disorders Group seemed like a statement that ME/CFS was a mental disorder.

Given the complexities around the causes and physical impact of ME/CFS, however, there wasn't a precise match with any Cochrane editorial group. So the Cochrane Editorial team sought a more general “fit" in both topic and approach, including enthusiasm for improving the review and working with stakeholders. Taking on a contentious review with a troubled history, and the negative attention that brings with it, is a major commitment.

The review's new “home" isn't a specific review group, which is a first for a Cochrane review: it's Cochrane's MOSS Network. Acronyms for the groups involved in these editorial arrangements are:

  • MOSS: The Cochrane Musculoskeletal, Oral, Skin, and Sensory Network;
  • PaPaS: The Cochrane Pain, Palliative and Supportive Care Group;
  • EMD: The Cochrane Editorial and Methods Department.

This pairing of review and subject area probably isn't the first candidate to spring to mind. However, this Network's subject remit and background gives it particular affinity with this project, including reviews in conditions such as fibromyalgia and chronic pain, and with extensive experience in stakeholder participation. This review is a distinct project, though, independent of other reviews.

One of the MOSS editorial groups, the PaPaS review group, will help shepherd the review through its search strategy, quality assurance, peer review, and publication, in collaboration with MOSS and Cochrane's EMD. The scope of PaPaS is broad, incorporating supportive care for people living with serious illness, and chronic pain. The team is based in Oxford, UK.

The project manager for the update is Rachel Marshall, from the EMD in London. In addition, two of the authors, including the lead author, come from the Cochrane Response team – the team is introduced below. The Cochrane Consumer Engagement Officer, Richard Morley, will be available to support members of the IAG and consumer authors. And the Cochrane communications team will coordinate any media enquiries.

There are occasional joint editorial meetings about the review involving MOSS, PaPaS, the EMD, and me. You can read about the policies and processes of PaPaS here. One exception to these processes for this review will be a brief public consultation of the peer-reviewed protocol. A preprint of the manuscript will be released, and a consultation process will be planned by the IAG. As well as submitting written comments, there will be opportunities for people to comment using less formal communication methods.

The review itself will follow the usual PaPaS processes. MOSS and PaPaS will make the final choice of peer reviewers for the protocol and the full review, including consumer peer review, with input from the IAG. The IAG will continue for some time after publication to consult and advise on responses and concerns about the published review. The formal commenting system for Cochrane protocols and reviews will be available.

3. Review authors

The new author team was decided by MOSS and PaPaS, in consultation with the Cochrane project manager and me. The IAG was not yet established at that time. We agreed on this composition for an author team of eight:

  • Two consumers;
  • Two clinicians with experience of clinical care with people with ME/CFS – one medical, one physiotherapy;
  • Two systematic reviewers with relevant methodological expertise – including a statistician; and
  • Two systematic reviewers from the Cochrane Response team, including the lead author.

Consumers, as defined by Cochrane, could be people with personal experience of ME/CFS or carers for them. We agreed on these criteria for these positions:

  • Different experiences/beliefs about exercise and ME/CFS that reflect something of the range of people's experiences, and ideally different experiences of ME/CFS;
  • A history of interest in research on ME/CFS;
  • Experience working in groups producing work with people of different views, shown by having previously been involved in science-related groups or a background of working in a professional environment where processes involved collaboration;
  • An understanding of the implications of being involved in an ME/CFS controversy, demonstrated by having been public about their experience with ME/CFS, and using social media; and
  • One with a background of ME/CFS community advocacy and one without, and at least one woman with ME/CFS, as ME/CFS affects predominantly women.

We aimed for geographic and gender diversity in the team.

As mentioned above, after Cochrane posted in March 2020 that additional new authors were being considered for this review, there were expressions of interest. We are grateful to all of them for their interest, and all were considered as potential authors. In addition, the project manager identified a shortlist of candidates for clinician positions via a search of recent relevant publications and other scientific and community involvement.

I nominated several candidates for the consumer author positions, with some consultation, again considering expressions of interest, plus searches of social media and community activity, and consumer representatives in committees addressing ME/CFS issues in scientific organisations. I also nominated candidates for the authors with specific expertise, after reviewing expressions of interest and seeking some recommendations.

After people accepted the invitation to join the review team, their disclosures of interests were reviewed by PaPaS, in accordance with Cochrane policy.

You can see biographies and disclosures of interests of the authors here. They bring a great depth and breadth of expertise to the project, and we are grateful to them for their commitment:

  • Consumers: Mary Dimmock (USA, carer, person with ME/CFS advocacy background), Kay Hallsworth (UK, person with ME/CFS);
  • Clinicians: Todd Davenport (USA, physiotherapy), Julia Newton (UK, medical);
  • Systematic reviewers with relevant methodological expertise: David Nunan (UK, experience with systematic reviews of exercise as an intervention); Argie-Angeliki Veroniki (Greece, statistician);
  • Systematic reviewers from Cochrane response: Hanna Bergman (Israel), Nicholas Henschke (Germany, lead author).

4. Independent Advisory Group (IAG)

The IAG has now been largely recruited. The goals of the newly formed group are to:

  • Ensure and facilitate responsiveness to stakeholders' concerns; and
  • Improve the relevance, accessibility, and credibility of the revised and updated review.

The proposed composition of this group with a total of 10 people, was announced in March 2020. We received multiple expressions of interest and nominations of individuals and stakeholder groups. We are grateful to all of them for their interest, and all were considered as potential members. We will be forming a discussion group of community/stakeholder groups to provide an additional communication channel.

The positions were filled in several stages, aiming to achieve gender balance and geographical diversity, as well as membership with credibility to a variety of stakeholders. The decisions were mine, in liaison with Cochrane.

To achieve a balance of independent appointment and community representation, for the three positions representing people with ME/CFS internationally I invited organisations to hold these positions: the participating individuals are their nominees. Although there is no direct Cochrane policy for this, we asked the three organisations for informal disclosures of interest: we did not consider having an established policy on exercise and ME/CFS a conflict of interest. For individuals, we adopted a process on conflicts of interest similar to that for Cochrane authors.

Two of the 10 positions are currently vacant, and will be filled soon: one will be appointed by the IAG itself, and the other is a position to replace a systematic review methodologist who had to withdraw because of clinical pressure due to the pandemic. You can see biographies and disclosures of interest of the members here.

IAG membership (eight appointed, two vacancies):

  • Nominees of three organisations representing people with ME/CFS:
    • Penelope McMillan (with Penelope Del Fante, and proxy, Geoffrey Hallmann) (Australia) - ME/CFS Australia;
    • Jaime Seltzer (USA) - #MEAction;
    • Samuli Tani (Finland) - European ME Alliance;
  • Two people from a clinical background with active interest in ME/CFS:
    • Lily Chu (USA) (medical practitioner);
    • Peter Gladwell (UK) (physiotherapist);
  • Individuals who had been involved in discussions with Cochrane preceding the IAG:
    • Hilda Bastian (Australia, Lead);
    • Mike Clarke (Ireland/UK, systematic review methodologist); and
    • George Faulkner (UK, independent researcher).

Communication to date has been by email and individual discussions given the time zone spread, and that will largely continue. However, there will also be some formal online meetings.

You can read more about the remit of the IAG here.

Update note: New description of ME/CFS advocacy on 4 June 2021 - Hilda Bastian.