Heather Bagley (COMET), Sarah Gorst (MRC Hub for Trials Methodology Research), Richard Morley (Cochrane) Bridget Young (COMET) explore how patient-imporant outcomes could be identified for Cochrane Reviews. 

An enthusiastic group of healthcare consumers, Cochrane authors,  and researchers joined a workshop jointly run by Richard Morley (Cochrane Consumer Co-ordinator) and Sarah Gorst (Research Fellow for the Patient Perspectives Theme of the MRC North West Hub for Trials Methodology Research) during the UK Cochrane Centre Symposium in Keble College, Oxford 14-15 March 2017. The aim was to explore how patient-important outcomes could be identified for Cochrane Reviews and to answer these specific questions:

• What are the challenges of gaining a patient perspective about patient-important outcomes in Cochrane Reviews?
• How might core outcome sets (COS) help in a Cochrane Review, and what are the benefits and challenges in introducing a COS for reviews?
• What happens when a relevant COS does not exist? How can we seek patient input to ensure that the outcomes used in reviews are important to patients as well as to clinicians and researchers?

Background

Ill health and treatments can affect people in many ways, making it difficult to select the most appropriate outcomes for research. The Cochrane Handbook emphasises the importance of consumer involvement in decisions about outcomes for reviews, whilst also promoting the use of standardised core outcomes and says review authors “... must ensure that there is effective public and patient involvement to identify the patient-relevant questions for Cochrane Reviews and patient important outcomes.” (training.cochrane.org/handbook)

This is even more important in the light of the recently published ‘Evaluation of NIHR Investment in Cochrane Infrastructure and Systematic Reviews’, which called for “more involvement of Cochrane Consumers, and other stakeholders in the whole systematic review process” including in “question formulation, scoping, outcomes, and dissemination products  (beyond  the  Plain  Language  Summary)”  as a way to “improve  the  relevance  and  uptake  of reviews.” (www.journalslibrary.nihr.ac.uk/nihr-research/evaluation-of-nihr-investment-in-cochrane)

So, what’s the problem with outcomes in research?

There are a number of challenges associated with the use of outcomes in research.  Different studies that look at the same treatments for the same condition often measure different outcomes.  When such studies are completed, we cannot compare or combine their results in systematic reviews and meta-analyses because they have used different outcomes.  The scale of this untidiness is striking. For example, over 25,000 of the outcomes in cancer trials have only been used once or twice (Hirsch 2013). Also, in a case study of Cochrane Reviews of HIV/AIDS, Saldana (2017) reported that outcomes used in trials and reviews were not well aligned.

The presence of selective reporting of outcomes in trials, which occurs when a study team set out to collect information on several outcomes, but publishes the results for only some of the outcomes. This can happen when the findings for particular outcomes differ from those for which the team hoped. Smyth (2010) found empirical evidence of outcome reporting bias, and calculated that 29% of trials displayed it.  Dwan (2008) also showed that outcomes that are statistically significant are more likely to be fully reported, and that studies that report positive or significant results are more likely to be published.

The outcomes used in research and systematic reviews may not be relevant or important to patients. Numerous studies have reported differences in the outcomes identified as important when patients or carers have been involved, for example, Noble and Marson (2016) explored health professionals’ views about outcomes of importance in relation to epilepsy.  When further work was conducted involving patients and carers, they found that whilst the patients and carers agreed with the outcomes identified by the health professionals, they also identified three additional outcome domains of relevance that included depression, anxiety, and independence/need for support. Similarly, Turk (2008) compared the outcomes identified by patients with those identified by health professionals in the area of pain, and reported that, “In addition to confirming the importance of pain relief and improvement in physical and emotional functioning, our results expand on the [IMMPACT] domains by highlighting fatigue, sleep, home and family care, social and recreational activities, interpersonal relationships, and sexual activities.”

What solutions might there be?

If all studies investigating a particular health condition used the same outcomes, they could all be compared and combined. This would reduce research waste by making best use of all the data, and would also reduce selective reporting bias. When a set of main outcomes has been agreed for a health condition, it is called a ‘core outcome set’ (COS).  A COS is “An agreed standardised set of outcomes that should be measured and reported, as a minimum, in all clinical trials in specific areas of health or health care” (COMET: www.comet-initiative.org). Patient input into agreeing outcomes of importance is vital.

How are core outcome sets agreed?

COS have to be relevant to patients, carers and health professionals and other key stakeholders, so the people developing COS need to ensure that the expertise of these groups is used to agree the core outcomes.  To seek such agreement researchers often use ‘consensus methods’ such as the Delphi process which involves a survey and a final consensus meeting to agree which outcomes are the most important, i.e. the ‘core’, outcomes (for further information about the Delphi process see: www.comet-initiative.org/assets/downloads).  Some researchers also conduct interviews with patients to understand their perspectives about important outcomes.

What is the COMET Initiative and how is it helping?

COMET stands for the ‘Core Outcome Measures in Effectiveness Trials’ Initiative. This initiative involves people from around the world from many different areas of health and social care, and was set up to help the development of COS in two main ways:

• firstly, by providing the COMET database (www.comet-initiative.org/studies/search) as a central point where researchers, practitioners and patients could find out which COS that have already been developed or are still under development; and
• secondly, to provide materials to support teams who are working on COS.

Using COS

To date, the following organisations have endorsed COMET:

• Arthritis Research UK (ARUK) www.arthritisresearchuk.org;
• Health Research Board (HRB) www.hrb.ie;
• The IDEAL Collaboration www.ideal-collaboration.net;
• National Institute for Health Research Health Technology Assessment Programme (NIHR HTA) https://www.nihr.ac.uk/about-us/how-we-are-managed/boards-and-panels/programme-boards-and-panels/health-technology-assessment/;
• Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) www.spirit-statement.org.

According to Professor Hywel Williams (Chair of the NIHR HTA Commissioning Board), “Patients and professionals making decisions about health care need access to reliable evidence. The…COMET database will help researchers across the NIHR family and beyond when choosing the outcomes to include in the studies that will establish this evidence base.”

Increasingly COS are being identified as part of the Cochrane Review process and Section 5.4.1 of the Cochrane Handbook (‘Listing relevant outcomes’) states: “… several clinical areas are developing agreed core sets of outcome measures for use in randomized trials, and consideration of these in defining the detail of measurement of outcomes selected for the review is likely to be helpful.”

How do we involve patients when there isn’t a COS available?

The literature that describes the involvement of consumers in identifying outcomes in systematic reviews is limited. While evaluating the experience of public research partners involved in a systematic review, Vale (2012) identified the importance of consumers being “involved at the stage when the questions are being developed to give them the opportunity to influence which outcomes should be investigated”. Morley (2016) recently sought to identify the extent of consumer involvement in systematic review production within Cochrane. His team found a number of published studies that described consumer involvement in setting outcomes that are important to patients: including Kelson (1999), which identified case studies from the Pregnancy and Childbirth and Musculoskeletal review groups; Ghersi (2002), which identified the impact of consumer involvement in the Breast Cancer Group; and Coon (2015), which reported end-user involvement in a systematic review of quantitative and qualitative research of nonpharmacological interventions for attention deficit hyperactivity disorder delivered in school settings.

Additionally, recent work commissioned by the UK Cochrane Centre and led by Sally Crowe, explored ways of identifying cost effective, robust and evidence-based outcomes that involved stakeholders in in asthma, chronic rhinosinusitis and breastfeeding (training.cochrane.org/resource/CLL-webinar-consumers-terrified).

Meanwhile Cochrane’s Project ACTIVE (training.cochrane.org/ACTIVE) is bringing together evidence, information and resources about the involvement of people in systematic reviews so that Cochrane authors can access published examples of ways of involving people throughout the systematic review process (Pollock 2016).

Nevertheless, there is no consensus on the best methods for involving patients in the Cochrane  Review process where an appropriate COS does not exist, or, in cases where a COS does exist, for choosing outcomes, for example, for inclusion in the ‘Summary of findings’ table where a maximum of seven outcomes is required and the existing COS contains eight or more outcomes.

Workshop discussions

The presentations at the workshop (as mentioned above) were followed by a lively discussion in which participants explored the benefits and challenges of patient input into the selection of outcomes for reviews; considered how the use of COS could help; and potential methods to use when employing the patient perspective to select patient-important outcomes for reviews where an relevant COS currently does not exist.

The workshop participants identified the following benefits of involving consumers:

• ensuring the relevance of systematic reviews to patients by using more relevant outcomes and keeping research grounded in people’s real life concerns;
• identifying outcomes that clinicians and other stakeholders might not have described;
• enhancing public engagement with research;
• increased ownership of the results by patients, which leads to increased uptake of research findings; and
• giving patients the opportunity to make a valuable contribution to research.

Several challenges in involving consumers were voiced, including:

• funding and capacity issues;
• difficulties in identifying patients to engage;
• ensuring diversity (especially of seldom-heard populations);
• difficulties in resolving diversity of opinions and priorities;
• challenges that may arise if the identified outcomes are difficult to measure;
• potential conflicts between patients’ experience of illness and previous received knowledge of a condition;
• difficulties in communication between patients and clinicians; and
• the lack of tried and tested methods that could be used.

What next?

There was considerable enthusiasm among participants about the possibility of involving consumers in identifying outcomes in systematic reviews. However, it was also agreed that more research was needed to identify effective methods for doing this, in order to provide guidance for review authors.

In the meantime COMET has published its new Handbook that can be accessed at www.comet-initiative.org. The COMET People and Patient Participation, Involvement and Engagement (PoPPIE) Working Group has produced resources for researchers seeking patient and public input into COS development (www.comet-initiative.org/ppi/researchers). ‘Involving People - A learning resource for systematic review authors’ is a new resource for Cochrane authors who want to involve consumers in Cochrane Reviews will be launched in September 2017 and will be available at training.cochrane.org.

Acknowledgements

The content of some of this article is based on the COMET Core Outcome Set Plain Language Summary (www.comet-initiative.org/assets/downloads).

References