William (Bill) Silverman (1924-2004) was one of the founders of American neonatal medicine. He was honoured repeatedly as one of the pioneers in his specialty; however, he often evoked somewhat contradictory responses amongst his colleagues because he was in the habit of raising troubling questions about the scientific basis and ethics of his and their practices. Like many of the people who have helped to establish Cochrane, Bill Silverman could be regarded as a 'troublemaker'. As he reiterated frequently, however, criticism is a form of troublemaking that can help to drive progress. Furthermore, criticism should not be limited to examining the work of others, but should also include self-criticism.
Every year Cochrane recognizes the outstanding work of Bill Silverman with its own award. It is offered annually and explicitly acknowledges Cochrane's value of criticism, with a view to helping to improve its work, and thus achieve its aim of helping people make well-informed decisions about health care by providing the best possible evidence on the effects of healthcare interventions.
Livia Puljak won for her paper; ‘Risk of bias assessments for blinding of participants and personnel in Cochrane reviews were frequently inadequate' We asked her about this award and what it means to her.
What made you decide to enter for this prestigious Cochrane award?
A long time ago, I was talking to Dario Sambunjak, a fellow Cochranite, about my frustrations with inconsistencies in the risk of bias (RoB) assessments in Cochrane Reviews, and inconsistent advice I was getting from colleagues when conducting my Cochrane Reviews. I told Dario that I started analyzing a large sample of Cochrane Reviews to assess the prevalence of RoB judgments that were not in line with recommendations from the Cochrane Handbook. Dario laughed and said – you will get Bill Silverman Prize one day for this. That is when I first heard of the Prize. In our study, we evaluated one aspect of the preparation of Cochrane Reviews, and we provided constructive suggestions on how the relevant aspects of Cochrane’s work could be improved. Thus, our team truly feels that our work can have a positive impact on the scientific quality of future Cochrane Reviews.
Can you tell us a little more about what made you decide to enter for the Prize and the process involved?
The published paper, for which we received the prize, aimed to analyze whether judgments about RoB for blinding of participants and personnel (performance bias) in published Cochrane Reviews were made in line with instructions from the Cochrane Handbook for Systematic Reviews of Interventions. Our first author, Dr. Ognjen Barčot, designed a computer code that extracted data from Rob tables in Cochrane Reviews. We analyzed judgments for performance bias of 10,429 randomized controlled trials (RCTs) that were included in 718 Cochrane reviews. Overall, 1,828 out of 6,918 judgments (26%) for performance bias were not in line with the Cochrane Handbook. In reviews where Cochrane authors have split the performance bias domain into two subdomains, based on blinded individuals (one subdomain for blinding of participants and another subdomain for blinding of personnel), we found lower prevalence of inadequate risk of bias judgments – 9% of judgments for blinding of participants, and 5.8% judgments for the blinding of personnel subdomain were judged inadequately. We concluded that interventions should be considered for improving the adequacy of RoB judgments in Cochrane reviews; these interventions can be made at the level of editors, peer-reviewers, and authors. Motivated by these results, we even designed our own RCT in which we tested one intervention for improving the adequacy of RoB judgments of RCT reports.
When we do Cochrane Reviews, critical appraisal is an important part of the research process. However, we should apply the same scrutiny to Cochrane Reviews too – when we conduct Cochrane Reviews, we can notice inconsistencies and opportunities for improvement, and these can be translated into formal methodological studies with the ultimate aim to improve new Cochrane Reviews. This is what motivated me to design this study.
How did it feel to win? What does this award mean you, personally?
I was very proud and honored when I got the news that our publication got the Bill Silverman Prize 2020. I am also immensely proud of our research team, and I would like to thank colleagues Ognjen Barcot, Matija Boric, Svjetlana Dosenovic, Tina Poklepovic Pericic, and Marija Cavar for participating in this study and making it happen.
I am truly impressed that Cochrane is recognizing the value of criticism coming from methodological studies (also called research-on-research studies, or meta-research studies). In my personal experience, such studies are often underappreciated, and disregarded as “not a real research”. However, I never let such comments discourage me. Constructive criticism of research practices is the way forward, enabling learning and improvement. If we do not take a step back to analyze what we do, how we do it, and how we can improve it, we will never make a progress. Thank you, Cochrane, for this award.
- Read Livia Puljak's winning paper; Risk of bias assessments for blinding of participants and personnel in Cochrane reviews were frequently inadequate