This Blog presents commentary and personal opinion on topics of interest from a range of contributors to the work of Cochrane. Opinions posted on the Cochrane Blog are those of the individual contributors and do not reflect the official views or policies of Cochrane.
It was sobering to reread the following blog written in March 2016. On one hand, I recognised that all too human trait for dispensing more gung-ho advice than one would necessarily apply to oneself. And on the other hand, I realised that a zealous application could have prevented the publication of some high quality reviews on key topics. Even so, the underlying messages remain pertinent to this next phase of Cochrane. And I anticipate that reading this will be liberating, particularly in conjunction with the recently published Rejection of Cochrane Reviews policy. Ours is not the only CRG that has provided disproportionate support to authors of poor quality reviews. This approach has engendered running backlogs that have persisted for years. There have been disheartening endgames for several priority reviews that still did not make it to publication despite extensive feedback and support. Thus, I resolve to be stronger and tougher. I hope this following will help others avoid some of the pitfalls when editing Cochrane Reviews.
Helen Handoll
Co-ordinating Editor
Cochrane Bone, Joint and Muscle Trauma Group
January 2018
What to do on finding a review in need of attention.
1. Shout for help!
A good bellow is always worth a try. Although you may not find professional help (a statistician or even a content expert) readily available - demand greatly outstrips supply – making someone aware that you are about to edit a review is highly recommended (see tip 11).
2. Assess the situation
Do not plunge straight into the review. A few quick checks on the circumstances may prevent or lessen despair in the long run (see tip 11). Always check it is the most up to date version. Sometimes it might be worth someone asking the review authors if they had actually meant to submit their review. Beware though that the euphoric feeling upon discovering that the review does not need to be saved may be short-lived.
3. Check the protocol
Locating and looking at this key document is a crucial step. The attachment of review authors to their protocol varies but can be severely weakened when they are in the mixed and dangerous company of primary research reports. Carefully laid out inclusion criteria and methods can be cast aside. Study selection may seem haphazard but sometimes you start to realise there has been a change, which may even be announced in the text, in the review’s scope. Pause before taking precipitative action, because it is possible the authors might be right (see tip 11). More commonplace is some mild forgetfulness of intent and purpose. Perusal of the methods section may find superseded methods blithely recast in the past tense, and a source of false claims.
4. Out for the count (the numbers game)
An initial illusion that a review is fine may be chipped away with small discoveries of inaccuracies (how many trials were there again?) and inconsistencies. Be prepared for a plunge in your estimation of the authors’ ability, diligence and attention to detail when you check the information and data extraction from one of the included trials. The dismay is often renewed when you check a second trial report. The old adage “why ruin a good story with the facts” comes to mind; but such discrepancies should not be taken lightly. There is no point in looking further than the results or providing a full or comprehensive feedback report in these circumstances.
5. Mind the language
When reading the text, always remember that you are there for the reader of the published review. If you do not understand it or find yourself lost in a busy stream of disorganised text rescued only by the next section heading, be assured you will not be alone. However achieved, a rewrite usually together with some restructuring will be necessary. Uneven quality in language may, regrettably, be a sign of plagiarism or poorly paraphrased text. Ironically, helping the authors improve such text may result in a greater similarity with the original copied but unattributed text.
6. Try to avoid being taken for a ride
As an editor, you end up doing quite a lot of thinking and more for the authors. Naturally, there are times when you wonder why you are spending hours of your limited spare time providing feedback when it is abundantly clear that some if not most of the authors have contributed rather less than you have. Indeed, one suspects the honest answer to “have all the authors actually read the review” will be “No!” It is likely there will be freeloaders, who were perhaps on nominal standby to arbitrate if necessary, in powerful positions that make it awkward for the working author(s) to call on their help or even content expertise. You may even find that you are acting in a proxy PhD supervisor role, while the real supervisor is proving hard to get but might be spurred into action to ‘support their student’ where there are excessive editorial delays holding things up. One telltale sign of zilch involvement is where these aloof review authors are also trial investigators on included trials but have not resolved queries on their trials. From experience this can stretch from failure to clarify trial methods to failure to notify on duplicate publications. It is inevitable that you will feel a mug from time to time but you should give yourself permission to throw in the towel where an extreme effort is required and just send it back to the authors. Ultimately an abject review should be their problem not yours.
7. Feedback – getting the message across.
Take advantage of proformas that help you structure your feedback. Never fool yourself that you will remember all the points you wished to make. Keep to the point in your feedback: review authors are rarely interested in lengthy explanations. Avoid giving more than one instruction at a time. Avoid sweeping assumptions of the authors’ diligence – for most authors addressing feedback is a necessary evil, not an end in itself, and authors will often comply with specifics not generalities. Authors generally don’t ‘get’ irony. Try to find something good to say about the review, perhaps at the start, and maybe, provided it does not stretch credulity, at the end. Give as realistic estimate of the work entailed as possible.
8. Temper the tone
Whatever your mood after your dispiriting ordeal in trying to rescue a review - whether sunk into despair, vexed to the point of screaming, or fermenting with barely restrained fury – the tone of your feedback needs to be calm and respectful. Remember, with rare exceptions, at least one person on the byline has worked hard to complete their review. Consider also whether there are redeeming features worthy of remark. Enlisting a diplomatic intermediary, to screen and smooth the prose, can be advisable. Otherwise, set aside and revisit before dispatch.
9. Wait
This can be the hardest part as by handing over your feedback and revised review to the review authors, you have relinquished all control. It is prudent to recognise that all your effort may be in vain – indeed the shock and outrage that anything was found wrong with their review can be too much for some authors – or that intensive remedial work may again be necessary. Whatever, you should be prepared for a long wait. On looking through authors’ response documents, it is especially important to avoid the illusion that authors understand the word “done” in response to direct instructions. Be wary also of finding much comfort in repetitive expressions of gratitude; “thank you” may instead anticipate the review authors’ perception of your future industry and continuing benevolence.
10. Time for reflection
A struggle as it may be, try to avoid getting cross with authors. Remember that some reviews cannot be saved and that, for these, ‘battling on’ is not the best option. Also remember that editors have rights too. Contemplations on this last sentiment will provide material for a future piece on ‘How I reclaimed my life’. The race is on…
11. Editors are not always right and sometime make mistakes
Helen Handoll
Co-ordinating Editor
Cochrane Bone, Joint and Muscle Trauma Group
(Originally published March 2016, updated January 2018)