Is there value in making author contact when determining risk of bias in a Cochrane Systematic Review?
Helen Atherton is a Research Fellow in the Department of Primary Care Health Sciences at the University of Oxford, and a Cochrane author with the Consumers and Communication Review Group. She is a health services researcher and her main research interest is in eHealth, particularly the use of information and communitcations technology (ICT) in healthcare delivery. In this post she discusses the value of author contact in assessing risk of bias of studies included in Cochrane Reviews.
When conducting a Cochrane Review, attempts to assess risk of bias are often restricted by incomplete reporting of study design, e.g. not reporting on allocation concealment. When information is missing we end up rating domains as ‘unclear’, which is unhelpful for review authors, editors and readers. One approach is to contact the authors of the included studies and ask them for the missing information on study conduct. The Cochrane Handbook for Systematic Reviews of Interventions advises that though this is an option, contacting authors of trial reports may lead to overly positive answers, so a careful approach should be taken in doing so. Overall, little evidence is available about how well author contact works for assessing risk of bias.
We completed three Cochrane Reviews in 2012, all evaluating the effects of using email in healthcare settings. In each review the majority of included studies had not provided adequate descriptions of key aspects of study design, such as method of randomisation. A communication method such as email is a complex intervention, and testing it in the context of a trial presents challenges. Aside from the complexity posed by variation of format, use of email can take different forms, and is used differently across different settings and populations. Additionally, with complex interventions, the write-up of studies becomes more challenging: context is influential in how a complex intervention works, so authors often need to convey detailed contextual information in a short journal article format. Information may be left out, by accident or design.
Owing to these factors, reviews of complex interventions look very different to reviews of, for example, devices or drug treatments. The complexity was reflected in our reviews by the differences observed between the included studies. Faced with included studies that looked so different from each other, and realising that the review would present a narrative synthesis rather than a meta-analysis, we felt that getting a good assessment of risk of bias seemed vital in allowing us to appraise the studies clearly that we did have, so we decided to contact the authors. We figured that it was worth a go in trying to learn more about the studies and what their findings might mean.
We sent each author a standardised letter, using a template provided by our Review Group (Consumers and Communication), together with a list of the type of risk of bias on which we wanted to obtain information. Bearing in mind that some studies had been published some time ago, we also provided a brief definition of the type (commonly known as the domain).
Contacting authors worked well for our reviews. We were able to obtain contact with all the authors, via email, with the exception of one author from a study published in 1995, whom we managed to contact via letter. Review One (Email for clinical communication between patients/caregivers and healthcare professionals) included nine studies, each featuring unclear domains. Before author contact, 34 of 54 domains were unclear; after author contact only three of 54 domains remained unclear. Review Two (Email for the provision of information on disease prevention and health promotion) included six studies; five had unclear domains. Author contact reduced the number of unclear domains by 50%, from sixteen to eight. Review Three (Email for clinical communication between healthcare professionals) included only one study, and author contact changed the number of unclear domains from two to one. The risk that study investigators would offer overly positive answers did not hold true for our reviews. We actually considered that the information they provided made us more inclined to rate the risk of bias as ‘high’ rather than ‘low’. A full breakdown of the risk of bias assessments and their changes before and after correspondence will be reported in full in an oral presentation at the Cochrane Colloquium in Québec.
These findings may reflect the nature of the studies. Trials of interventions such as email are a relatively new development: authors tend to be enthusiastic clinicians, rather than drug companies or established research teams. The fact that our response rate was so high also indicates that much of the information we needed was already available directly to the study investigators. Maybe the motivations for conducting the research are different - or perhaps there is less riding on the results?
Our findings did change our perception of the included studies. Where before we had been uncertain about how reliable they were, the risk of bias information was used along with other factors (choice and measurement of outcomes, implementation of interventions) in deeming the findings of the included studies largely unreliable. In view of this, is author contact something that should be advised for all reviews of complex interventions? It is possible that the value of author contact lies in the nature of the intervention under review. Most interventions under review are not as complex as email, so this approach may not offer as much in terms of context.
The narrative approach to the review meant that we did not have to use risk of bias assessment in deciding whether or not to exclude studies. For reviews with meta-analysis, the risk of bias information may be used in a different way: to determine which studies have been included in the analysis, or to determine whether an analysis has been conducted. Where analysis is not possible, the risk of bias information becomes useful in a different way, providing material with which to synthesise in a narrative way.
How might Cochrane guidance help authors to address these issues? Making author contact for risk of bias assessment optional does not lead to consistency across Cochrane Reviews, as it is left to the discretion of the review author. If there is potential for author contact to have an impact on the conclusions of a review it should be mandatory, and consistent methods should be developed. Otherwise, authors may be introducing bias in deciding not to do this - though this is arguably easy to say when reviews have a low number of included studies. Authors of reviews with included studies in double figures and above may disagree. Maybe there would have to be a cut-off? More than one unclear domain in 50% of your included studies? Contact authors. Dealing with a complex intervention? Contact authors. Where should the line be drawn? Other things to bear in mind are restrictions due to time and funding. Many authors are already preparing reviews as extra workload beyond their day jobs: is there capacity for another stage in what is already an arduous process?
Amidst the fog of reviews that were complicated and presented a challenge in how to present the data, author contact allowed for some clarity on study conduct. It was valuable. How valuable it could be as part of the wider reviewing process is currently a known unknown that should be addressed.
Helen Atherton (firstname.lastname@example.org)