Dr. Vaibhav Sahni holds dual specialisations in the fields of Oral and Maxillofacial Surgery as well as Periodontics. He also holds the MFDS RCPS (Glasg), MFDS RCSEd and the AssocFCGDent. His areas of expertise include, in addition to the general purview of oral health, maxillofacial injuries, point of care diagnostics, oral-systemic interactions, hostmicrobial interactions as well as computational systems as they relate to oral health and rehabilitation. He is currently a postdoctoral scholar at the Forsyth Institute, Cambridge, Massachusetts, USA. He is a member of the Cochrane Early Career Professionals Network. Here he looks at how and where systematic reviews are registered. 

Registration is an important aspect of conducting a systematic review as it helps in reducing research waste, decrease bias and enhances the overall transparency of the work (See here).

Registering a systematic review involves publishing details related to the project prior to its commencement. This enables one to verify whether the proposed methodology and reporting criteria were met after the review is concluded. 

In addition to these reasons, it makes sense to register a protocol and authors are no longer bound by the lack of free registries, as there are now many places to register. Some authors might be of the opinion that registering their protocol might lead to their ideas being hijacked by some other research team making their work irrelevant. On the contrary, authors should view this exercise as registering a claim on their idea and workflow. Another potential issue authors might think of is that developing a protocol is a time-consuming task which basically is of help to other reviewers. This process should be viewed in light of the fact that systematic reviews are conducted by more than one person and having a set protocol avoids confusion, miscommunications and bias within the team. Another advantage of an a priori protocol publication is that it strengthens confidence in the eventual report when the workflow has been followed, it conveys that the authors did not alter the work to suit their preferences. 

Some organizations such as Cochrane, which commission or conduct their own reviews maintain an exclusive registry of their published and ongoing work. From being merely an idea in Janury 2010, currently, the prospective registration of systematic reviews seems to have become an established practice. 

From an author’s perspective, a prospective register needs to:

a.    Allow authors to prospectively register their systematic reviews.
b.    Provide an overview of currently registered systematic reviews.
c.    Enable a search function for the sake of being user-friendly.

Duplication in conducting reviews is a significant issue. 3 It would serve authors well, to search registers and databases to ascertain whether their research is in fact new and not identical or similar to ones already out there.

Below is a summary and overview of some of the more popular prospective registers to house a systematic review protocol.

PROSPERO, INPLASY and Research Registry are specific for systematic reviews, while protocols.io and OSF Registries are generic in that they accept all study types. The systematic review registers have a structured registration form usually consisting of 24-28 fields which are mandatory. OSF Registries allow users to either select the ‘Open-ended-registration’ option or from varying forms, the former allows for authors to provide a narrative description of the review. All registers have the facility to search from registered systematic reviews, the optimization and utility of these tools, however, varies. PROSPERO tends to provide the most elaborate search options, but recent reports have deemed the search tools suboptimal (See here). 

PROSPERO, however, helpfully guides users through the process of navigating their process.

The registries also vary in terms of their eligibility criteria to accept entries. A case in point is PROSPERO which requires the systematic review being registered to possess a health-related outcome.

When making a decision regarding which registry to utilize, a significant point of contention would be to weigh the processing time against the cost of the service. A service such as INPLASY would charge authors but aims to provide faster turnaround times as compared to a service such as PROSPERO, which is free of charge, but has longer turnaround times due to its popularity.

Another issue with a popular service such as PROSPERO is that it is funded by the United Kingdom (UK) National Institute for Health Research (NIHR) which means that UK-based registrations are dealt with on a priority basis. This may become problematic for researchers outside the UK wishing to register.

There are also some other ways to attempt transparency in one’s methodology. Online data repositories such as Figshare, OSF and Zenodo, enable users to upload project-related data and documents which are tracked for their version, have a time-stamp and are assigned a DOI. re3data.org, a collection of research-based data repositories may also be searched to find other such platforms.

Some other options to look into may include the AHRQ Systematic Review Data Repository, CAMARADES (Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies) and AsPredicted.

It appears that prospective registries for systematic reviews are fast becoming an essential component of the research process for a variety of reasons. Authors would find that investing some time in making the effort to register their projects would go a long way in aiding their research endeavour by eliminating research waste, enhancing transparency and decreasing bias.
 

- Dr. Vaibhav Sahni

• Learn more about Cochrane Early Career Professionals Network
• Early Career Professionals Network blog post: Learn more about GRADE

Cochrane Group members are spending between 3 hours to just 15 minutes a week on social media activities. How can you make the most out of that time for maximum impact? In this blog Roses Parker, Research Fellow, and Muriah Umoquit, Communications and Engagement Officer, share communication tips that are easy to implement and take minimal time.  

Often the people doing social media are for a Cochrane Group are doing it along side other responsibilities. To help with this capacity crunch, we've gathered our 5 best hacks to save you time but get the maximum output for any minutes you are able to invest!

1. Tap into the larger communication community on Cochrane Slack

What:  Core Cochrane staff are using Slack as their virtual office - if you are doing comms for a Cochrane Group, then you should be there too! In that workspace there is a 'channel' where suggested tweets are shared, graphics and top tips are posted, and you can ask questions and get help from others doing communications.
Why: It's easy to scan the chat in our channel for items that would be of interest to your audience and copy-and-paste relevant suggested tweets. And the biggest thing - you'll be part of a larger supportive community! 
Time commitment: 10 minutes every week.
Ongoing benefit: Once you have mastered the weekly scan and disseminating of suggested tweets, you can also use the channel to 'boost the signal' on your own news or ask questions. 
Links: Learn more about accessing Cochrane Slack and then head over to #all-comms-help

2. Use a social media posting scheduler 

What: An external tool that lets you manage your social media accounts from one place, allowing you to schedule posts at a later time. 
Why: You only have to sit down once a week and schedule all your social media posts at one time; these posts will automatically go out when you want, without you having to log-in.
Time commitment: 1 hour initial investment and then 10 minutes a day or less if you pair this with tip 1!
Ongoing benefit: Once you get the hang of scheduling, you can look at changing the messaging slightly so you can post about one review/topic multiple times for different audiences, time zones, or branching out into other social media platforms.
Links: We recommend Buffer (It's free for 3 social media channels and 10 posts at a time) and Cochrane UK has an helpful how-to video. 

3. Know your Dissemination Essentials

What: A checklist of 18 considerations when disseminating health evidence. There is a 1-page checklist that is great to print-out and keep handy; the full document is also available with some helpful examples.
Why: If you're low on time, then you are likely not going to be creating elaborate dissemination products but this checklist is fantastic to use as a reference when creating messaging for social media posts. By using the checklist, you will get information to the people who need it, when they need it, in a way they can understand and use it. 
Time commitment: One time commitment of 10 minutes to review the 1 page checklist and one hour to browse and review the supporting materials.
Ongoing benefit: Once you get familiar with the 18 items on the checklist and how to apply them, it can become more of a routine to communication messages. 
Links: Print the 1-page checklist and review the Dissemination Essentials supporting materials - a recorded webinar and the full document with examples. 

4. Keep track of how to reach your target audience

What: Create and maintain an Excel table that you can refer to that has common hashtags you can use to reach your audience and relevant organizations/association/advocacy groups/partners to tag. 
Why: You are likely to have more than one audience and it's likely not everyone from those audiences are following you on social media. Use hashtags and other trusted organizations to help get information you are trying to disseminate in front of the people who need to see it. 
Time commitment: 1 hour for initial set-up and a few minutes a week to update.
Ongoing benefit: Think of this as a long-term project you can build upon. It will become a personalized resource when planning out future dissemination.
Links: Start by looking at the Altmertic Scores of some of your recent top reviews (linked to on the review!) to see who is sharing your reviews and what hashtags they are using. You can also look up specific healthcare hashtags on Symplur.

5. Add a picture to your posts 

What: Add a relevant photo or graphic to your social media posts - Cochrane has a Dropbox of folder of helpful images which are free for you to use. 
Why: It's generally known that images get more attention on social media over not having an image - a small-scale study using Cochrane social media accounts showed this too! An 'Image Checklist' will help give you the confidence to select images and the Central Executive Team can help provide images to you for free. 
Time commitment: One time commitment of an hour to review the check-list and supporting materials and then an few minutes extra for each post.  
Ongoing benefit: You can tailor the pictures you use to your target audience demographics so that the pictures used are relatable and appropriate to your setting and culture. This may concern ethnicity, skin colour, gender roles, hierarchy, moral ideas and common behaviour, gestures, religious practices and places, food, meaning associated with colours or symbols, and design preferences. 
Links: View the Image Checklist with helpful examples and learn about getting access to a free Dropbox folder of images.

We hope you put some of these 5 hacks into action! If you found this helpful, take a look at these other communication blogs:

• 5 tips for a low-maintenance Cochrane group website
• Social media in 10 minutes a day

Cochrane’s revised policy on conflicts of interest (CoI) was launched in October 2020. Since then, Cochrane’s Research Integrity team has been working with authors and editorial teams to support the implementation of the policy and has been learning about the challenges that arise. One such challenge has been the suggestion of authorship changes by authors and editorial teams as a way to comply with the CoI policy. This blog post from Stephanie Boughton, Research Integrity Editor in Cochrane’s Research Integrity team, addresses why such authorship changes are not a solution to CoI policy breaches and what should be done instead.

The CoI Policy for Cochrane Library content (2020) differs from equivalent policies in many journals in that, as well as requiring declarations, it also limits or prohibits involvement in a particular Cochrane Review for individuals with certain conflicts.

Conflicts that entirely prohibit involvement in a Cochrane Review include employment by a commercial organization with a financial interest in the topic area of the review, or personal ownership of a relevant patent. Others, such as payments from commercial organizations with a financial interest in the topic area of the review, prohibit being first or last author.

The policy also places restrictions on the author team as a whole, with a requirement that two-thirds are free of all relevant financial CoI. Finally, there are also restrictions on what a review author who is also an author on a study that may be eligible for inclusion can do in relation to the review. 

Collectively, these measures aim to minimize CoI and maintain the highest levels of objectivity and user confidence in content published in the Cochrane Library.

In the 18 months since the policy launch the Research Integrity Team and CoI Panel have received a variety of CoI referrals and queries from Cochrane Review Groups and review authors as the policy has been applied to new protocols, reviews and updates. Among these, two recurring issues stand out – firstly how related CoI and authorship are, and secondly, the importance of collecting CoI declarations early in the process, before any work has begun, and ensuring that authors are aware of the importance of not gaining further relevant CoI while working on the review.

When are authorship changes not a solution to CoI policy breaches?
In situations when conflicts resulting in a breach of the CoI policy have come to light at a later stage in the review process, after work has begun, we have received questions about whether changes to authorship could resolve the issue.

An example is when a first or last author has declared a relevant CoI during the review process, and we have been asked if changes can be made to the order of authors, moving the conflicted author to an allowable position as a middle author.
 
We have also seen cases where there have been requests to remove a conflicted author from the author team after they have already worked on the review, either so that the author group meets the requirement for two-thirds of authors to be free of all relevant financial CoI, or so that someone with a CoI that prohibits involvement in Cochrane Library content is not listed as an author. 

In all these examples, changes in authorship are not a way to comply with the CoI policy because such changes do not remove any potential effect that the CoI may have already had on the review.  Instead, changing the authorship serves only to mask any effect the CoI may have had, and may be misleading. Furthermore, authorship changes at a late stage do not adhere to standards for authorship described below.

In these cases, to comply with the CoI policy, the review must start over with a new author team that complies with the policy, highlighting the importance of checking CoI early in the process.

The importance of collecting CoI declarations early
Under the CoI Policy for Cochrane Library content (2020), CoI declarations are collected at multiple stages before and during the review process. This starts with the collection of declarations of interest from all proposed authors before any work begins (at title registration, or before the decision is made to update).

Collecting and checking potential CoI at this early stage, coupled with review authors being made aware from the start of the importance of not gaining further relevant CoI during the review process, and the consequences of doing so, helps prevent problematic situations from arising later in the review process.

Authorship
As with collecting and checking CoI early on in the process, agreeing authorship and position in the author line up early on ensures everyone is aware of their role on the review and can prevent potential issues, such as authorship disputes, arising later in the process. 

In addition to Cochrane following guidance from the Committee on Publication Ethics (COPE) on authorship, such as obtaining written agreement from all authors for any changes in authorship after work has begun, we have also recently updated the authorship policy in the Cochrane Database of Systematic Reviews editorial policies to clarify that authorship changes, including changes in author order, are not a solution for resolving CoI breaches.

Further help and advice
Resources to help with applying the policy are available for Cochrane Review Groups and authors on the CoI portal, including answers to FAQs from Managing Editors and a quick guide to the CoI Policy for Cochrane Library content (2020) for authors.

Cochrane’s Research Integrity Team is also available to answer any queries about the CoI policy – either via our CoI referral process for those relating to a specific review, or via email to coiarbiter@cochrane.org for more general queries.

You can also learn more about the research projects the Research Integrity Team is involved in here. If you would like to get involved or have any suggestions for relevant projects, we’d love to hear from you at researchintegrity@cochrane.org.

Stephanie Boughton

Research Integrity Editor
Cochrane’s Research Integrity team

Dr. Luis Enrique Colunga-Lozano is both an internal medicine and a critical care physician from Guadalajara, México. He currently works at the COVID-19 intensive care unit at Hospital Civil de Guadalajara - Dr. Juan I Menchaca. He is also a junior professor at Universidad de Guadalajara, where he teaches evidence-based medicine. Dr. Colunga-Lozano received his Master’s degree in Health Research Methods from McMaster University under Prof. Gordon Guyatt’s supervision. Furthermore, he has been a member of Cochrane since 2013 and has been involved in the production of systematic reviews as part of various Cochrane Groups including the Metabolic and Endocrine Disorders Group, the Heart Group, and the Schizophrenia Group. He is also a member of the Geographical Group in Guadalajara, Mexico. Dr. Colunga-Lozano works closely with members of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group on guideline development projects.

The assessment of the certainty (quality) of evidence is an important task during evidence synthesis, with implications on evidence applicability in decision-making. Several methods are available to evaluate the certainty of evidence. However, the GRADE approach has become the preferred method due to its systematic process and transparency. This Cochrane Blog provides a summary for researchers that are not familiar with the GRADE approach. A more detailed description is available in the GRADE series published in the Journal of Clinical Epidemiology. Useful resources are also available on the GRADE working group’s website. Please visit: https://www.gradeworkinggroup.org/

What is GRADE?

The GRADE working group is a collaboration that began in 2000, with an interest of addressing the limitations of grading systems in health care. The GRADE working group developed a sensible and transparent approach to assess the certainty of evidence and strength of recommendations. To date, more than 110 organizations have adopted the GRADE approach as part of their methodology (e.g., WHO, UpToDate, Cochrane, DynaMed, the National Health Service, and the Centers for Disease Control and Prevention).

The first article describing the GRADE approach was published in 2004 in the British Medical Journal (BMJ). In 2008, the BMJ published a series containing guidance for clinicians to understand the GRADE approach, and from 2011 to the present day, more than 30-parts of a series expanding the GRADE concepts have been published in the Journal of Clinical Epidemiology, which aims to provide guidance on systematic reviews and clinical guideline development.

What are we grading?

There are two components that we rate when we are using the GRADE approach. Firstly, the certainty (quality) of the evidence, and secondly, the strength of recommendations. GRADE divides the certainty of the evidence as very low, low, moderate, and high confidence. It is important to remark that GRADE does not intent to assess single studies, it is about assessing bodies of evidence, ideally produced from systematic reviews. GRADE also classifies the x

What are the GRADE criteria for assessing the certainty of evidence?

Currently, there are specific criteria for assessing certain bodies of evidence (e.g., treatment (pairwise), network, diagnosis, and prognosis). For example, in the assessment of treatment effects, randomized controlled trials will start as high confidence, and observational studies will start as low confidence. However, confidence in the evidence can change based on the following domains. Evidence certainty will be downgraded by the following: risk of bias (-1, -2), inconsistency (-1, -2), indirectness (-1, -2), imprecision (-1, -2) and publication bias (-1, -2), and upgraded with the following: large effect (+1), dose response (1+), all plausible confounding (1+).

What are the GRADE criteria for assessing the strength of recommendations?

Recommendations are informed by the evidence and a strong recommendation implies that benefits outweigh the risk/hassle/cost. There are several determinants of the strength of a recommendation. We can consider three as very important: Firstly, the balance between desirable and undesirable effects. For example, if the desirable consequences outweigh the undesirable consequences, there is more likely to be a strong recommendation, or if the desirable and undesirable consequences are closely balanced, there is more likely to be a weak/conditional recommendation. Secondly, the higher the quality of the evidence, the higher the likelihood that a strong recommendation is warranted. Finally, if the guideline panel is very certain of the trade-off between the desirable and undesirable consequences of treatment, that means that the preference is similar between patients, there is more likely to be a strong recommendation. If the guideline panel is uncertain about patient values and preference, and they believe that this will be different across patients, there is more likely to be a weak recommendation. Other considerations include cost, importance of the problem, acceptability, feasibility, and equity (Evidence to Decision Frameworks).

What is the interpretation of a strong vs weak recommendations?

With a strong recommendation, the variability of patient preference will be less (i.e., almost all (>90%) of informed patients will decide on the same choice); whereas with a weak recommendation, informed patients will select different choices. With a strong recommendation (based on supporting evidence), interaction with the patient is to ‘inform’ the suggestion (e.g., I consider this should be the course of action, because of this…); whereas with a weak recommendation, the right choice may differ between patients, highlighting the need for a shared decision-making process to identify the right choice for each patient. Strong recommendations are considerations for quality of care, and weak considerations are not, because the right choice differs from patient to patient.

Conclusion

Healthcare professionals and policy makers need evidence summaries to help them during their decision-making, these summaries should report the quality of the evidence and the strength of recommendations. The GRADE framework provides explicit guidance on the assessment of the certainty of evidence and strength of recommendations. The GRADE approach is transparent and systematic, which may be one of the reasons it has been adopted by many international institutions.

• View Cochrane Training materials on GRADE
• Learn more about Cochrane Early Career Professionals Network
• Early Career Professionals Network blog post: Registering Protocols of Systematic Reviews

Jennifer Dawson, PhD has worked with Cochrane as the Wikipedia in Residence since 2016. Her role includes maintaining and building further relations with Wikipedia, connecting new editors to the Wikipedia community, and supporting requests for engagement in Wikipedia work from the Cochrane community. Here she shares a few of her top FAQs on how to fix what people are reading on the internet! 

Cochrane has been working with Wikipedia since 2014 to try and improve the health-related evidence that is presently being shared in the over 36,000 medical articles on English Wikipedia. Here are the top three questions that we get. 

I am interested in improving a health-related article on Wikipedia. How do I get started?

The best way to start improving Wikipedia is to register for an account and then go in and fix that mistake. “Being bold” is actually encouraged on Wikipedia! When ‘fixing’ a mistake be sure that your improvement is supported by a high-quality citation that meets Wikipedia’s guidelines! When you are ready to add more health-related evidence, visit the Cochrane-Wikipedia Project Page for more information.

I have found a Cochrane Review that may be helpful to use as evidence in an article. What is the next step?

I usually suggest that you spend some time reading the present version on the Wikipedia article first before jumping in with new ideas. This ensures that the evidence is not already shared in a different part of the article. Once you have a place where you feel that your evidence would be ‘useful’ to readers, here are a few tips:

• Paraphrase all evidence (we cannot use quotations from sources on Wikipedia)
• Be sure to integrate your evidence into the existing text. Often this means expanding on the existing background information section in the Wikipedia article.
• It is not appropriate to share the PICO details or other information about how the systematic review was conducted in the Wikipedia article. If people are interested in this information they can read the source.
• It is also not usually necessary to share the source of the evidence in the text of the article (e.g, please not note write “A 2021 Cochrane review found that….”) See below for an example of how to share evidence, taken from https://en.wikipedia.org/wiki/Cardiac_surgery.
• Pulling in other high-quality secondary sources in addition to the Cochrane review is also ideal.

BEFORE:

AFTER:

Example of an article improvement sharing background information from two different sources.

What if I have a conflict of interest in a field that I would like to improve on Wikipedia? What if I am an author on a review that I feel could be used to improve a Wikipedia article?

Conflict of interest is taken very seriously on Wikipedia. If you are an author on a paper or have a conflict of interest in an area that you would like to improve on Wikipedia it is great that you are asking this question! The first thing to note is that being transparent about your COI on Wikipedia is very important. Here are some ideas for how to get started:

• All Wikipedia articles have a “talk page” or back page where proposed content can be discussed. If you have a conflict of interest I encourage you to first start a discussion on the talk page and disclose your COI. More information about the talk pages can be found here: Wikipedia: Talk page guidelines.
• It is usually possible to post your ideas for improving on the talk page and then you can work with the existing volunteers to improve the article.

For more information please see: Wikipedia: Conflict of interest and Help:Wikipedia editing for medical experts or contact the Cochrane Wikipedia Project.

Jennifer can be found on Wikipedia at: JenOttawa and by email at jdawson@cochrane.org. Please visit the Cochrane-Wikipedia Project Page for more information.

Jennifer Dawson, PhD has worked with Cochrane as the Wikipedian in Residence since 2016. Her role includes maintaining and building further relations with Wikipedia, connecting new editors to the Wikipedia community, and supporting requests for engagement in Wikipedia work from the Cochrane community. Here she shares how Wikipedia works, what challenges exist...and how you can get involved too! 

Wikipedia for health information
English Wikipedia includes over 38,000 articles related to human health that receive over 2 billion page views each year from around the world. Articles on Wikipedia vary widely in page views and in quality. The majority of errors on Wikipedia have been found to be errors in omission, however, misleading or inaccurate medical content can be easily found.  

Wikipedia articles are created by volunteers, many of whom have registered Wikipedia accounts. Wikipedia volunteers include individuals interested in a particular subject including people with lived experience, health care professionals, students, and other volunteers. Volunteers choose to be anonymous or disclose their identities.  In order to ensure that reliable content is added to articles, any material added that pertains to human health requires a citation and that citation must follow Wikiproject Medicine’s Guideline for Reliable Sources (WP:MEDRS). For academic medical writers who are used to writing for peer-reviewed medical journals, following this guideline can be an adjustment that can take some time and practice. Rather than cite the strongest or latest primary research study, The WP:MEDRS guideline requires that all evidence added to Wikipedia (even background information on a disease) be first summarized and reviewed by a secondary source, preferably a strong secondary source such as a systematic review or clinical practice guidelines. Expert knowledge or opinion that is not backed by a secondary source is not permitted on Wikipedia. Wikipedia articles are essentially a summary of high-quality secondary sources and are not mini-reviews of direct primary research studies.

How can Cochrane help improve what people are accessing on the internet?

Presently over 3,500 Cochrane Systematic reviews are cited in English Wikipedia articles. Evidence shared in the background and results section of Cochrane Reviews can be considered to fill an evidence gap or expand an article. Not all Cochrane Reviews are appropriate for Wikipedia and many require that the Wikipedia Editor first expand the present article using other high quality secondary sources before.

What impact does this work have?

Directing people to high quality evidence based sources on the internet is a challenge. Millions of people are accessing health information via Wikipedia each year, but at least we can improve what they are accessing! Examples of top viewed pages on Wikipedia:

December 8, 2021 Wikipedia daily and monthly page views

How can I get involved?

Editing Wikipedia is fairly straight forward to learn. New editors are encouraged to start small. Finding a typo in an article and correcting it is a great way to get started. Expanding the background information in an article with paraphrased evidence shared in a high-quality text book or the background section of a systematic review is another great way to get started.

Check out the Cochrane-Wikipedia Project Page for more information on how to edit and to find Cochrane reviews that may be helpful to improve the present content on Wikipedia.

Jennifer can be found on Wikipedia at: JenOttawa and by email at jdawson@cochrane.org. Please visit the Cochrane-Wikipedia Project Page for more information.

Katherine Jones, NIHR Network Support Fellow, looks back to the Cochrane-funded ‘Running on  Empty’ project led by Paul Montgomery and the Empty Reviews Project Group, more than a decade ago. Having initiated and completed an update evaluation in 2021, she considers findings in the context of the original project. Read more about the update project here, and the report’s top 3 ideas for future evidence production.

Sustainability is a key focus for Cochrane as the organisation looks to a new model of evidence production that addresses the global challenges of today and tomorrow. Reflecting on earlier evaluations, such as the ‘Running on Empty’ project could help us to learn lessons from the past as Cochrane navigates its way forward. Exploring how evidence production has changed over the last decade could also provide a new perspective on what a more sustainable future might look like. As such, the purpose of this update project, completed during Katherine’s NIHR-funded position with Cochrane, was to encourage more discussions about the value of empty reviews and their contribution to future evidence production. 

So-called ‘empty reviews’ are those reviews that report no included studies. Katherine’s updated evaluation of the Cochrane Database of Systematic Reviews revealed a similar proportion of empty reviews about health topics as the original evaluation, and there were still substantial variations between Cochrane Review Groups. Meanwhile, the overall trend for first publications of empty reviews appears to have changed over the last decade. Read the full report here.

So, what are the top 3 ideas for future evidence production? 
The update evaluation report suggests to 1) revisit the importance of the review questions in empty reviews in consultation with review authors and key stakeholders 2) consider creating a filter for empty reviews in the Cochrane Library to help identify evidence gaps and inform prioritisation. And  finally, 3) consider new streamlined formats for publishing empty reviews.

• Read the full report

Rachel Craven, Product Lead of Cochrane Library, recently incorporated user experience research to help improve the right-hand navigation beside reviews in the Cochrane Library. Here she shares more about that project and offers her 3 top tips that you can incorporate into your projects at Cochrane. 

At Cochrane we want to make sure that our evidence is accessible and usable to support healthcare decision making. For the Cochrane Library, we wanted to simplify the right-hand navigation to improve user engagement with the content and improve the user experience of the website.

To support this work we undertook user experience (UX) research. UX research is the systematic study of target users and their requirements, to add realistic contexts and insights to design processes. UX researchers adopt various methods to uncover problems and design opportunities. For this project we interviewed a number of members representing our key user groups and asked them to complete tasks using two potential designs and talk us through the process while sharing their screen.

Given all the work needed in the backend, it was great to verify that the changes would be helpful and that we got things ‘right’ before launch! With the help of our user testers (our grateful thanks!) we ended up with this new right-hand navigation that has:

• Clearer call for action to key user tasks
• Prominent article metrics that better highlights content value
• New ordering of navigation that better reflects editorial content hierarchy
• Better showcase of dissemination/related products
• Use of icons to improve scan-reading

This new navigation has just launched in the Cochrane Library on all Reviews
– we welcome you to check it out!

Cochrane strives always to include feedback from end users in our evidence creation. We can also do that too when designing products or making improvements. Here are three tips that you might consider with your next UX project.

1. Add it into your project timeline: Add working directly with users into project planning early. For this project, we gave ourselves a month to conduct the interviews and go over the data.
2. Know your audience and seek them out: It’s great to head into the project with an idea of how many users you want to work with and what characteristics you are looking for. Cochrane Engage is the easiest way to find volunteers! We also put out the call on social media and sought help directly from Cochrane’s Early Career Professionals group.
3. Make it fun and easy for your participants: Sometimes it’s hard for people to explain what they like about a design or how they use things. Using a method such as onserving tasks or showing more than one design can help people to better articulate what works and doesn’t for them.

We hope you head over to the Cochrane Library and take a look at the shiny new right-hand navigation that has built with user input! Feedback on the Cochrane Library product, product strategy and product development is always welcome:

• Cochrane Library product feedback form
• Cochrane Engage | Cochrane Library: tell us what you think

The Cochrane Pain, Palliative and Supportive Care (PaPaS) Review Group was awarded a Cochrane Network Innovation Fund to produce expert guidance on how to manage pain as an outcome in systematic reviews of postoperative care. Co-ordinating Editor Dr Neil O’Connell and Managing Editor Anna Erskine from the PaPaS Group are pleased to present the results of this project – a set of resources to help drive improvement in the standard of systematic reviews in this field, and also a useful guide for those seeking to use systematic reviews to guide clinical decisions.

Background and introduction
We probably all think about the potential for pain when we think about surgery. Pain is a common primary or secondary outcome in systematic reviews of perioperative anaesthesia and postoperative care. How methods are applied and outcomes are managed varies substantially across published reviews and this can present a number of challenges when trying to interpret their findings. Indeed, while it can appear to be simple, managing pain as an outcome presents us with a number of interesting challenges.

In Cochrane PaPaS, we are in the business of synthesizing evidence from clinical trials (and other well conducted studies) for the treatment and management of pain. In this process we have to make numerous decisions about the place of pain and analgesia in both the trials and the reviews.

Variability, inconsistency, and errors in pain reporting have been identified by Cochrane UK, by the Musculoskeletal, Oral, Skin and Sensory (MOSS) Network, and by other stakeholders. It is an area for which harmonisation and standards will help improve communication, impact, and therefore the reputation of the Library. We viewed this project as an opportunity to share good practice and expertise.

To develop tools/ guidance to support review authors dealing with acute postoperative pain in systematic reviews (in particular Cochrane Reviews).

What we did
We established an expert group of editors and authors, and consulted with the MOSS Network and the Cochrane Anaesthesia Review Group. We reviewed the evidence to support best practice, elucidated examples, and provide some recommendations and considerations. Discussion is presented on areas of uncertainty where further research is needed, how to incorporate this practice into systematic reviews, and on transferable learnings to systematic review practice in general.

The resources
All resources are freely available on the Methods website: https://methods.cochrane.org/methods-cochrane/reviews-post-operative-pain

From our meetings and discussions we identified the key themes of:

• trial accuracy and reliability;
• participants pain status and severity;
• measurement issues and important outcomes for pain;
• relevant associated outcomes of interest;
• analysis issues.

From these we developed a checklist of critical items that authors of reviews in this field should consider when designing and conducting their review. The checklist also serves as a guide to optimal methods for people who are trying to make sense of a systematic review that has postoperative pain as an important outcome.

The checklist is accompanied by sets of slides and reference lists for the following topic areas:

1. some background to postoperative pain and its importance;
2. measurement of pain emphasising the importance of patient reporting;
3. outcomes in pain trials emphasising patient-reported outcomes of value to people with pain;
4. fundamental considerations in acute pain studies;
5. issues of interpretation of evidence around small studies, and small numbers of small studies in postoperative pain;
6. an examination of dealing with publication bias used in acute pain studies;
7. a worked example of possible sensitivity analyses that can be used in reviews of postoperative pain.

Benefit and impact
We hope the resources will reduce variability and inconsistency across Cochrane Reviews in the Library, and provide the community with the opportunity to improve their understanding of clinical trials investigating acute postoperative pain.

Beyond supporting Cochrane authors, we also hope these resources will be useful and used by Cochrane editors, authors of non-Cochrane Reviews, and anyone using Cochrane Reviews of interventions for acute postoperative pain to make informed decisions about healthcare in this area.

We welcome any feedback or comments on these resources at any time, so please do contact the team directly if you have any suggestions or would like to discuss any aspect of the resources.

Acknowledgements
Thank you to the Cochrane Network Innovation Fund for investing in this project.
Thank you to Mohammed A. Abusayed (University Hospitals of Derby and Burton, UK) for auditing reviews of interventions for pain in the Cochrane Library in 2016.

Thank you to all the project team members, advisers, and MOSS key contacts.

• For any queries, please contact support@cochrane.org
• View the 'Optimal methods for the use of ‘pain’ as an outcome in systematic reviews of postoperative pain management' resources

The full report of a Cochrane-funded study designed to compare the existing and revised Cochrane risk of bias tools for RCTs was finalised in June 2021 and has now been made available to the Cochrane community. The study was initiated to collect evidence about the amount of time and effort review authors need to invest to use the revised risk of bias tool for RCTs (RoB 2) compared with the existing study-based tool (hereafter referred to as RoB 1), and how review quality is impacted. The work, led by Bernd Richter and Bianca Hemmingsen from the Cochrane Metabolic and Endocrine Disorders Group, represents an important step in assessing the benefits RoB 2 holds for improving the reliability and quality of Cochrane Reviews, and whether there is a compromise with time taken to complete the assessments.

RoB 2 represents a change to the way bias is assessed and reported in Cochrane Reviews and was designed to guide authors through meaningful, results-based assessments. The revised tool requires a change in focus from studies to the data they contribute, and biases introduced into review analyses. However, following concerns about the usability of the revised tool and the time it might add to the review process, Cochrane set out plans to support the first author teams and review groups using it in a pilot. The aim was to observe any difficulties encountered throughout the review journey and build resources and functionality in our technology that would minimize the disruption of wider uptake. The comparison study ran alongside the pilot to compare RoB 2 with RoB 1 and collect empirical evidence within an index review.

The study was conducted using data from a Cochrane review on insulin analogues for type 1 diabetes mellitus, and the review sample for comparison was 26 RCTs that included parallel, crossover, non-inferiority, superiority, open-label and blinded designs. The review looked at eleven outcome measures including both dichotomous and continuous outcomes and objective and patient-reported endpoints. The team applied both tools to all outcomes according to a pre-specified protocol to compare key features, difficulties in use, and potential consequences for the results of the systematic review.

What were the main findings?
The study authors found mean assessment times using RoB 2 for all outcomes comparable to RoB 1 (approximately 30 minutes), despite experiencing a learning curve with RoB 2, and reported no major usability problems with either tool. Overall, the study found few difficulties with RoB 1 or RoB 2 in achieving agreement between raters and differences in consensus judgements between tools were not common. The biggest divergence between the tools was with subjective outcomes in open-label studies, where RoB 1 was more likely to penalize than RoB 2. RoB 1 also tended to lead to harsher risk of bias judgements as a result of the options available (high/low/unclear), whereas the ratings, algorithm, signalling questions and guidance built into the RoB 2 tool made it easier to work through complexity and context. The study also investigated the impact of clinical study reports (CSRs) for bias assessment with both tools and found them to be very valuable irrespective of the tool used, though the time taken to complete assessments was longer (45 to 90 minutes).

Differences were reported in the nature of discrepancies requiring consensus for RoB 1 and RoB 2. With RoB 1, disagreements were most common in the  ‘performance bias’ and ‘detection bias’ domains for participant-reported outcome measures. These biases are teased out in a more granular way with RoB 2, and the authors noted that systems to capture outcome-based variation with RoB 1 can overlook issues (e.g. by grouping objective or subjective), or require post-hoc clinical judgement that might itself introduce bias. For RoB 2, ‘deviations from intended interventions’ (domain 2) and ‘selection of the reported result’ (domain 5) led to most discrepancies, particularly when trial protocols were not available.

The study’s conclusions highlight the benefits of RoB 2 in encouraging reviewers to think in terms of endpoints and results to improve the quality and relevance of risk of bias assessment in Cochrane reviews. The authors also make several recommendations to improve usability of RoB 2, such as adding elaborations or tweaking wording for some signalling questions, and creating processes within the Excel® tool to automatically populate study-based data that is not expected to vary across outcomes.

What does it mean for Cochrane?
The results of any comparison between the two tools may be expected to vary with knowledge of the topic area, methodological expertise, familiarity with the existing tool, and the type and number of outcomes within the review. Therefore, no one study can provide definitive results about their relative usability and impact. The authors of this study are both experienced methodologists and familar with the clinical area. They also applied both tools in line with MECIR to capture differences in bias across outcomes, which may have reduced the extent of difference between the tools, both in terms of time taken to conduct assessments and translation to impact on the analysis results. Interpretation of the study’s findings must be made in light of these factors, and though the efficiency difference may have been starker had RoB 1 been applied at study level, it follows that so too would reliability and usefulness of the assessments.

The authors also highlight that differences in the way the tools are structured and how reviewers choose to embed bias judgements in their analyses limit the usefulness of some comparisons, especially when it comes to demonstrating impact on analysis results. Nonetheless, it is an important addition to the real-world evidence base and allows Cochrane to understand how the tools vary and continue tailoring implementation resources to address common issues and errors. Studies like the one reported here also provide rich stories to streamline how the tool is implemented through technology to create efficiencies and make it easier for the benefits of bias assessment to be realised.

It is in light of these considerations that the expectations for RoB 2 have remained as they were set in November 2020 , which incorporate an element of flexibility in continuing to use RoB 1, providing it is applied in a way that allows for differences in bias across outcomes to be captured. The expectations accommodate the ongoing learning curve with RoB 2 and circumstances where, on the joint judgement of authors and CRG staff, adopting the new tool might not be feasible (e.g. for a large update). This does not preclude that purely study-based bias assessments should be a thing of the past and that training, planning, careful consideration and debate within the author team should underpin any meaningful bias assessment.

The Methods Team and Bias Methods Group welcome feedback about the usability of RoB 2 and any suggestions about how the Excel tool and the wording of signalling questions can be improved, and will continue to develop resources to support the author teams and editors.

Attributed to Kayleigh Kew, not on behalf of EMD

Related links:

• Full report of the study  for more detail about how it was conducted and what it found
• News item stating expectations about RoB 2 in light of the comparison study
• Blog post summarising important resources and practical guidance for Cochrane authors and editors using RoB 2