Confessions of a rookie consumer peer reviewer

Cochrane Common Mental Disorders recently asked Karen Morley if she would give consumer peer review a go. In this blog, first published here, Karen provides an insightful and honest reflection of how she found the experience and notes important challenges that Cochrane and Cochrane groups can improve on.

Would I, as a new consumer volunteer for Cochrane Common Mental Disorders, have jumped in at the deep end and attempted a peer review if I hadn’t been asked? I don’t think so. But now I’m here, unsteadily dogpaddling, with two protocols and a systematic review under my belt, I thought I’d share some reflections and questions with you.

First came the Fear. I felt quite tiny in the huge, complex Cochrane organisation. And ‘peer reviewing’ sounded so imposing. I know there are consumers who are themselves researchers or health professionals, but how could I be considered a ‘peer’?

Jess Hendon, the Managing Editor, was warm, kind and welcoming and assured me my lived experience was what counted. But did I have the knowledge, understanding and skills for the task? I had some experience of close reading and critical evaluation from my years as a secondary English teacher, and a basic understanding of systematic reviews from contributing to some Cochrane consumer learning resources that are currently in development, but would that be enough? I was afraid of being judged and afraid what I had to say might not be useful. Well, it was too late for dithering; a deadline was approaching: I took the plunge.

There was a checklist that gave some focus and structure. There was also some linked online support. At the moment this is quite brief and packed with detail and assumes some prior knowledge in the reader (for instance, ‘interventions’, ‘outcomes’, ‘comparator’ and ‘settings’ have specialist meanings that might be unfamiliar to a complete newcomer) and the language is rather formal. I was lucky to have already explored the subject a little and I wondered what it must feel like to be starting from scratch.

Despite the checklist I was still full of doubt. What exactly was expected of me? How should I express myself and in how much detail? A worked example would have been helpful. When I later saw the feedback from the authors on all the reviewers’ comments I had a much clearer idea of the process and the kind of contributions that were acceptable. I felt strangely detached and lonely as I looked at my screen. It would have been nice to have the luxury of a mentor, or some buddies to bounce ideas off, or maybe to have had a group practice run. It would also be interesting to hear an author’s view. (I’ve since had the pleasure of meeting Lindsay Robertson, which I found reassuring – thanks, Lindsay!) Last year I took part in a James Lind Alliance Priority Setting Partnership and I found it was illuminating to hear the comments of other patients, carers and clinicians: we have lots to learn from each other.

The protocol I looked at first would certainly be challenging for a consumer to read. I had to check my own understanding of ‘monoamine neurotransmission’, for instance. (That was quite a ride!)

I’d be interested to know whether, when consumer reviewers write about language and style, the same issues crop up frequently, and whether it might be desirable or feasible to collect them or to have some sort of glossary. Making suggestions for explanations or alternative expressions throws up a host of knotty problems. Who is the reader I’m writing for? What level am I pitching at? Which consumers read protocols – and systematic reviews - and why? How much can something be simplified without losing important nuances or even completely misrepresenting its meaning? Is there in fact a conflict between several different audiences and purposes? Which ones do the authors have at the forefront of their minds? Why doesn’t a protocol have some kind of Plain Language Summary?

I suppose I felt the most confident about working on the language and less so about commenting on things like objectives and the criteria for considering studies. I knew there would be other contributors but I had a strong sense of responsibility: a real desire to write something that would actually be helpful to the authors. I have enormous respect for them and the huge, intricate, demanding task they have carried out. I don’t want to waste their time.

But now to the heart of the matter: representation. ‘Does the background address the hopes and concerns of people considering the treatment?’ asks one of the prompts. My task is to put myself into the shoes of the most important end users of systematic reviews and protocols: the consumers. This is the biggest responsibility of all. I can do my best but I’m aware that, alone, I’m not up to the task. I am one old, white, middle class woman, with one experience of one manifestation of a wide-ranging mental health condition: one tiny part of a huge jigsaw. Cochrane needs many consumer peer reviewers to represent us in all our diversity. And not just reviewers. I appreciate my opportunity to make changes to the review before it’s published, but this is the end of a very long journey. Have other consumers been involved before I see the review – in setting priorities or determining the research question or outcomes, say? How would I know? Shouldn’t it be mentioned in the review somewhere?

So, consumers, you are needed! If you feel the stirrings of an inclination to get involved, please have a go at something - anything. You might feel reticent, but come and help lead the way until it becomes commonplace for us to be included and to shape research around all our hopes and concerns.

I am sometimes shocked when I think how comparatively young the modern evidence based medicine movement is. Similarly, it’s easy to forget that the meaningful inclusion of consumers is really only beginning. I am delighted that Cochrane’s revised 2018 Peer Review Policy means Cochrane Review Groups must now aim to have at least one consumer peer reviewer for each Cochrane Review. (If there were more of us, fellow consumers, it could be more than an aim!) I’m delighted and daunted in equal measure to be one of those peer reviewers, and enthusiastic about CCMD’s plans to take this forward. It has been a tremendous pleasure to meet Rachel Churchill, Jess Hendon, Lindsay Robertson and Noortje Uphoff (well, I popped into her office!). The reception they have given me has been affirming and encouraging. I feel the burden, but I can see a great opportunity.

Karen Morley

6 May 2019