The National Institute for Health Research (NIHR) Complex Reviews Support Unit provides timely and appropriate support for the delivery of complex reviews that are funded and/or supported by NIHR - including Cochrane Reviews! Project Manager of the unit, Mora Aitken, answers a few questions to introduce the unit more widely to the Cochrane community.

What’s the Complex Reviews Support Unit and what do you do?
In July 2015, a consortium of researchers, led by Professor Olivia Wu, Health Economics and Health Technology Assessment (HEHTA), University of Glasgow, was awarded £2 million by the National Institute for Health Research (NIHR) to provide support within the UK in areas of research that require complex evidence synthesis. The NIHR Complex Reviews Support Unit (CRSU) is a collaboration between the University of Glasgow, the University of Leicester, and London School of Hygiene and Tropical Medicine.  CRSU is funded by NIHR to support and encourage successful delivery of complex reviews of importance to the UK NHS, whilst building capacity and capability within the research community through the support given.  

What are your aims and who can you support?
The aim of CRSU is to provide flexible, timely, and appropriate support for the successful delivery of complex reviews that are funded and/or supported by the NIHR, and in particular, reviews undertaken by Cochrane UK Groups.    

What support can you provide?
We provide one-to-one support on methodological challenges, as well as seminars, workshops, and training courses in evidence synthesis. Our expertise includes: diagnostic test accuracy reviews, network meta-analysis, individual participant data meta-analysis, economic evaluation, and realist synthesis and narrative synthesis of quantitative and qualitative data.  

What have you been doing for the past year?
To fulfil its remit, the CRSU has undertaken a programme of events to reach out to the NIHR research community and in particular the Cochrane community. This has been initiated by a series of workshops, the introduction of ‘Seminars with Cutting Edge Methods’ and individual support to Cochrane Review Groups (CRGs). Two successful workshops were held at The Cochrane UK and Ireland Symposium 2016 in Birmingham in March – ‘Methodological Challenges in Complex Reviews’ and ‘The NIHR Systematic Reviews Programme: Opportunities for Greater Impact’.   

What are your future plans?
The beginning of year two sees several initiatives in place to ensure that CRSU continues to increase its outreach to the research community to provide novel and sophisticated methodological approaches in synthesising different types of data. Planned activates include developing training courses, running further workshops in collaboration with individual CRGs and participating in the 2017 Symposium. In addition, CRSU will also be at hand to provide support to CRGs receiving awards from the NIHR Incentive Scheme and Programme Grants. CRSU anticipates a second challenging and fulfilling year, building a successful working relationship with Cochrane in supporting the UK NHS in delivering clinically and cost-effective services that are evidence-based.  

How do I apply for support?
For further information on CRSU and to apply for support, see our CRSU website.

_{The CRSU is funded by the National Institute for Health Research (NIHR)  (project number 14/178/29)}

Cochrane’s translation strategy, established in 2014, is aimed at strengthening impact in non-English speaking countries. More than 2,500 translations of Cochrane Review summaries have been published in the first half of 2016 alone. Hayley Hassan, Cochrane’s Translations Support Officer, offers an insight into the impressive work of Cochrane’s translation community.

Who are Cochrane’s translators?
Cochrane translators include health professionals, students, and translators. Located all over the world, they all have one thing in common: they are keen to help make Cochrane evidence accessible to speakers of their native languages. Local Cochrane groups usually lead the different language projects and support hundreds of volunteers, often dedicating a lot of their own free time.

Which languages is Cochrane evidence available in?
Our translation teams currently translate and disseminate our evidence in 13 languages:

• Croatian
• French
• German
• Japanese
• Korean
• Malay
• Polish
• Portuguese
• Russian
• Simplified Chinese
• Spanish
• Tamil
• Traditional Chinese

What do teams translate?
Since our Reviews are large documents, our teams usually translate the scientific abstracts and / or the plain language summary (PLS) sections. This year, some teams have expanded the scope of their translation activities, working hard to translate and record podcasts, as well as the blogshots developed by Cochrane UK. Translated blogshots have been tagged by language - check them out on the recently launched Cochrane Tumblr account!

How many translations have been published?
Thanks to the hard work of our teams, we have published 17,466 translations on our websites as of July 2016, along with more than 2,500 Reviews translated or updated this year alone. Elsewhere, 130 blogshots and 56 podcasts have been translated in 2016.  All of our teams’ achievements are summed up in the infographic.

How does translating evidence help us reach a wider audience?
Last year, more than 60% of the total visits to Cochrane.org were made using an Internet browser set to a language other than English, demonstrating just how global our audience really is. The increase in access to translated content on cochrane.org rose dramatically and consistently during 2015: By the end of the year, access had almost quadrupled compared to January. Access statistics for 2016 are very encouraging and we hope to continue to see this effect as more translations are published.

What’s new this year?
One of our main targets in 2016 and beyond is to improve access to the Cochrane Library for non-English speakers, and we are currently working to develop it into a multi-language website.

We strive to improve how we support and communicate with our translators. A translation toolkit has been created to support teams with their translation and dissemination activities, while a regular email digest provides them with relevant updates and reduces ad-hoc communication. In an attempt to help our translation project managers feel more connected, we run regular virtual meetings, and have recently started a Slack channel, to have a space to discuss and share experiences. The biggest challenge for volunteer teams is sustainability, so this year some of our teams received a small amount of funding from core Cochrane funds. The effect of the funding will become more apparent as the year unfolds, but our teams’ achievements and engagement have been tremendous already.

What can you do to support our translation work?

• Read more about getting involved in translation work.
• Follow our Twitter account, @CochraneLingual, and re-tweet updates to your followers.
• Cochrane Review Groups can promote translations of their work through social media.
• Take a look at the translation update in the weekly Communication Network digest, and promote what is of interest to you and your networks. Sign up here! https://lists.cochrane.org/mailman/listinfo/commsnetwork
• Subscribe to the translation mailing list for updates about translation projects and opportunities to contribute.
• Spread the word – share this blog post and infographic!

Hayley Hassan
Cochrane’s Translations Support Officer

Dario Sambunjak is a Learning & Support Officer at the Cochrane Central Executive and acts as the secretary to the recently established Cochrane Equity & Diversity Task Force. In this blog post, he discusses the first and last principles – “alpha and omega” - underpinning Cochrane’s work and invites Cochrane contributors to join in the effort of improving the goals of diversity, inclusiveness, and equity in the organization.

The spirit of working together in a welcoming, inclusive environment has always been the alpha and omega of Cochrane’s work. Literally. Just look at the 10 key principles underpinning Cochrane. The first one – ‘alpha’ – is collaboration. Cochrane built its success and reputation on co-operation and teamwork of many enthusiastic people across the globe. The last principle – ‘omega’ – is enabling wide participation. And how is this to be achieved? By reducing barriers to contributing and by encouraging diversity. Although much has been done in Cochrane to support this last principle, the words ‘diversity’ or ‘enabling’ are still far from becoming synonymous with ‘Cochrane’.

Principles are necessary guidance for persons and organizations as they grow and progress. But anyone who ever tried to live a principled life, knows that it’s not an easy feat. Which, of course, doesn’t mean that we should give up trying. The first step is to recognize that we are not fully up to the standard and explore the areas where improvement is possible. And that’s exactly the point where Cochrane is currently standing in terms of its ‘last’ principle.

Diversity and inclusiveness are explicitly mentioned in one of the four goals of Cochrane’s Strategy to 2020. In order to become a truly diverse and inclusive organization, Cochrane needs to ensure that involvement in its work, including production of systematic reviews, is fair and impartial. Or – to use another word – equitable. It is of note that Cochrane’s interest in issues around equity is not new. For more than a decade now, contributors of the Campbell and Cochrane Equity Methods Group have been active in putting the equity lens on the systematic reviews. Now is the time to use that same lens on Cochrane itself.
 
Some research and more anecdotal evidence indicate that in terms of engaging authors and other actors in the work of Cochrane, geographical and language background appear as some of the main sources of inequitable treatment within the organization. Other factors such as gender, race, and ethnicity also need to be considered in order to achieve the stated goals of diversity, inclusiveness, and equity.

The task is not easy and straightforward. Cochrane has to tackle underlying challenges such as the prevalence of dominant value systems and biases within its structures and ways of working. It also needs to be pragmatic and address immediately specific salient issues. To initiate these processes, the Equity and Diversity Task Force has been convened. However, it is clear that the work of improving equity and diversity in Cochrane cannot be entrusted solely to any small group of people. In order to engage a wider circle of interested people within Cochrane, the Task Force agreed to establish three Working Groups to review and address the following specific issues:

1. Title registration process, e.g., ensuring that titles are registered based on the quality of work and the transparently established priorities, and not on any other factor.
2. Other review production issues, including timeliness and nature of communications between Cochrane Review Groups and authors.
3. Other Cochrane equity and diversity issues, including those around translation, governance, communications and outreach, knowledge translation, and promoting greater diversity in Cochrane meetings.

The Working Groups will consist of the Task Force members and other Cochrane contributors sharing an interest in any of the above topics. Persons joining a Working Group are expected to volunteer some of their time and energy in designing and delivering actions to improve the equity and diversity around a specific issue. To achieve any success in this endeavour, we have to build and rely on the enthusiasm of individuals. Which is, by the way, the second principle – or ‘beta’ – on which Cochrane is based.

So, if you’ve read this blog post up to this point, it may well be that you are a suitable candidate to join an Equity and Diversity Working Group. If any of the three Working Group topics resonate with you and you want to get actively involved in this work, write to dsambunjak@cochrane.org. In your e-mail briefly describe who you are, your relationship to Cochrane, reasons why you’re interested in a particular topic, and any ideas on how you could contribute to a Working Group. Let’s see what we can do together in a welcoming, inclusive environment that is called Cochrane.

You may have noticed that the Altmetric ‘doughnut’ is now included at the start of every Cochrane Review and Protocol. But what does it mean? Working with Gavin Stewart, Cochrane Editor for our publishing partner Wiley, we’ve come up with all you need to know, including how authors, Managing Editors, and Review Groups can make the most of it.

What is Altmetric?
Alternative metrics provide a relatively new way of looking at the impact of published research and complement the traditional metrics such as usage and cites. In 2014, Wiley partnered with London-based company Altmetric to provide article-level metrics for more than 1500 journals and the Cochrane Library.

Altmetric track and report conversations and attention to research outputs from thousands of online sources, including mainstream news outlets, policy documents, and social media. Activity related to your Cochrane Review or Protocol that takes place on one of the many servers that Altmetric track is called a ‘mention’. The mentions are accumulated and weighted to give the published work an ‘Altmetric Score’.

What does Altmetric track?
Altmetric begin tracking mentions of your Cochrane Review or Protocol as soon as it is published in the Cochrane Database of Systematic Reviews. Examples of sources tracked for mentions include:

• Mainstream media: If your Cochrane Review is covered in the world’s press, it will most likely show up in your Altmetric score. Altmetric monitors over 2,000 outlets from around the world including Al Jazeera, BBC News, and The New York Times.
• Social media: Tweets on Twitter and posts on any public pages of Facebook as well as activity on Google+, LinkedIn, Sina Weibo, and Pinterest all add to your Altmetric Score.
• Blogs: More than 9,000 academic and non-academic blogs are monitored every day for mentions.
• Public policy documents: Policy documents from organizations such as the World Health Organization are monitored.
• Wikipedia: The English language version of Wikipedia is tracked for citations to published research.

It’s important to note that not all mentions contribute the same number to the Altmetric Score – the score is a weighted count. For example, a mention in the mainstream media contributes eight points, while a tweet contributes one point. More information on this here.

 Why is the Altmetric score important?
The Altmetric score can show Cochrane authors and Managing Editors if published research is being disseminated beyond the academic world. You can look at who is talking about Cochrane Reviews or Protocols, and make sure that they are being interpreted accurately and getting to the right audiences. Making sure Cochrane evidence gets to decision-makers globally is part of Strategy to 2020.

The Altmetric data can also be vital in securing grant funding and supporting promotion and tenure, as it shows the broader influence and impact of your work.

How can I make sure Altmetric pick up mentions of my work?
There are a couple of things that are required for Altmetric to correctly allocate a mention of your published research to the Altmetric Score of your Cochrane Review or Protocol:

• Always link to a page that includes your research’s unique identifier: In the case of Cochrane Reviews and Protocols, this is the DOI (10.1002/14651858.CDXXXXXX.pubX). Only mentions that include a link to the published version of the Cochrane Review or Protocol on Wiley Online Library will count towards the Altmetric score. A link to the abstract on Cochrane.org does not count towards the Altmetric score.  
• The link needs to be in the main body of the post: Unfortunately Altmetric can’t pick up any links included in headers or other sections of the page.

What if Altmetric have missed a mention of my work? How can I let them know?
The most common reasons that Altmetric might not include a mention in the Altmetric score of a research output is because a unique identifier was not present in the mention or the mention was made on a source that Altmetric do not track. If you think Altmetric have missed a mention of your work you can provide that information here.

How can I attract attention to my work?

• Plan early: Once your Cochrane Review is in the final stages of completion, it's time to think about making sure the right people know about it. You can find resources and help with this here.
• Share links to your work: Share via Twitter and other social media. Be sure to include relevant hashtags.  
• Start your own blog (or contribute to an existing one): It’s a great way to build your online profile and position your research.
• Reach out to key bloggers: Make them aware of your work. Look at the Altmetric details pages for other articles in your discipline to see who might be interested.
• Talk to Cochrane about Wikipedia: Cochrane and Wikipedia have a strong partnership; you can learn about the history in this video. You can learn more about getting your Cochrane Review on Wikipedia here.

I hope this provides a little more insight into alternative metrics and makes you curious enough to check out the Altmetric Score’s on some of your favourite Cochrane Reviews! You can learn more about ‘Altmetric for Researchers’ here. If you have questions, you can contact the Cochrane CEAD team.  

Download the summary below with working hyperlinks here

Jack Leahy is an Engagement Officer with Cochrane UK. Jack curates the new Cochrane Tumblr account, where the popular Cochrane blogshots are shared. These blogshots summarize a Cochrane Review with a visual and are easy to share and consume on social media – a perfect fit for the miroblogging platform that is Tumblr!

If you follow any of the Cochrane social media accounts, you will by now no doubt have noticed the growing success of our ‘blogshots’ which began as an initiative of Cochrane UK.  Blogshots are simple infographics that deliver key messages of a Cochrane review in a clear, consistent way and are perfect for sharing on social media.

There are now over 170 blogshots from Cochrane UK and an increasing number are being produced by other Cochrane groups. Blogshots have also been a popular way of sharing Cochrane evidence in other languages. Several of our translation teams have seen an increase in social media followers after sharing blogshots. Read more about Cochrane Iberoamérica’s experience here. This is why we have launched a new Cochrane blogshot archive, hosted on Tumblr, to ensure access for both Cochrane groups and the public.

How to use Tumblr

Check out the new blogshot archive on Tumblr here!

The blogshots are filtered by review group, series, and language if it has been translated. Just click on the categories and browse what is available! Each blogshot also has the review name, CD number, and URL to the Review or Plain Language Summery in the description. You can search the titles and CD numbers with the search function at the bottom of the categories.

If you see a blogshot you are interested in sharing, just right click the image to save it. The hyperlink to share with the blogshot on social media is included in the description.

How to contribute
If you have a review that you want to make a blogshot for, you can! We have templates and lots of guidance on doing this. You can read more about creating your own blogshots here.

Once you have a blogshot created, save it with name of the review and the CD number. Send the title, URL, and Review group with the blogshot image, the picture used, and the Powerpoint file (our translators may be interested in translating it!) attached to Muriah Umoquit. We’ll upload the new blogshot, tag it with the appropriate categories, and share with the masses!

Blogshots have attracted a lot of attention on social media –lots of shares and new people being exposed to Cochrane and evidence-based health care. Take a minute and browse through the new Tumblr account…I’m sure you’ll see something you’ll want to share further!

Jack Leahy
Cochrane UK

Related Resources:
Cochrane Tumblr
Blogshots – making evidence short and shareable for social media
Cochrane Translations: Cochrane Iberoamerica on translating blogshots
Cochrane Community Templates

It’s now two months since we launched Cochrane’s new citizen science platform, Cochrane Crowd, so it’s time celebrate our achievements and share the exciting plans for the months ahead.

Firstly a big thank you to our existing Embase screening project crowd for coming along on this transition with us and giving such helpful feedback on the early release of Cochrane Crowd – we’ve used your suggestions to tweak the platform and make it even easier to navigate for our new users.

Since launching Cochrane Crowd in March, almost 1000 new people have joined our crowd, over 250,000 classifications have been made and 6,000 RCTs identified for inclusion in CENTRAL – what inspiring numbers!

And not only that, we are rapidly approaching the incredible milestone of 1 million individual assessments made and over 26,000 RCTs identified since the start of the Embase project in 2014. Now that is something to celebrate.

Even more exciting are the plans for new Cochrane Crowd features and developments. Here’s a sneak peek of what still lies ahead.

Find out more:
Head to cochrane.crowd.org and start screening today, or read more about us on our website. And please keep sending us your suggestions about how we can make Cochrane Crowd better meet your needs.

Anna Noel-Storr
on behalf of the Cochrane Crowd team

Support for Project Transform was provided by Cochrane and the National Health and Medical Research Council of Australia (APP1114605). The contents of the published material are solely the responsibility of the Administering Institution, a Participating Institution or individual authors and do not reflect the views of the NHMRC.

The World Health Assembly is an important meeting for Cochrane and our partnerships. Here Sylvia de Haan, Partnership Coordinator for Cochrane, discusses what it is, our involvement thus far, and plans on making the most of our attendance.

Once a year the World Health Assembly (WHA) brings together 3,000 people representing 194 member states and more than 100 other organizations to discuss and debate global public health issues. The meeting runs for six full days and addresses topics such as health emergencies, childhood obesity, healthy ageing, nutrition, health systems, and infectious diseases, just to name a few. This is the place where global public health policy is discussed, debated, and agreed upon.

Cochrane has been in official relations with the World Health Organization (WHO) since 2011, and we have attended WHA meetings on and off since then. We have made statements in response to WHA agenda items, such as a statement this year on conflict of interest. This year we also organized, jointly with Health Technology Assessment international, a side event about evidence-informed decision-making and attracted around 100 people to that session.

However, it is probably not the involvement in formal events and WHA agenda items that make the difference. The WHA is all about networking. Ministerial delegations, WHO staff from headquarters and the regions, as well as civil society organizations, meet at the Palais des Nations in Geneva and get business done. A lot of connections are made, ideas brainstormed, and possible collaborations discussed.

‘During one single day I met with staff from various WHO departments and regional offices, with several member states delegations, and with other NGOs. This is a really unique venue for Cochrane to engage with a whole range of potential users of our work, and receive their input on their needs and priorities, and test how well we respond to these demands’, says Karla Soares-Weiser, Deputy Editor in Chief at Cochrane.

Till now Cochrane has been developing its WHA presence slowly. We have made a couple of statements over the years, connected to partners, and organized a side event. But shouldn’t we be doing more? And how can we make sure that you and your Cochrane Groups benefit directly from this investment? How can the Cochrane delegation best represent you all?

One great idea for the next WHA is expressed by Julie Wood, Head Communications and External Affairs at Cochrane: ‘While preparing for the Global Evidence Summit in Cape Town next year, we should really think about how we can use the 2017 WHA to connect to African delegations and enthuse them to participate in the Summit and share their needs and experiences in evidence informed health care. I am certain that this will greatly enrich our debates in Cape Town.’

We will certainly pursue this suggestion. If you have other ideas or would like to discuss how your Cochrane Group could profit from the WHA, please get in touch!

Sylvia de Haan
sdehaan@cochrane.org
Partnership Coodinator
Communications and External Affairs Department

David Roberts, a public health registrar with Cochrane UK, recently helped with a Cochrane stand at a Public Health conference. Here he shares a bit about the experience and gives tips for other groups sharing Cochrane Evidence.

The Faculty of Public Health Annual Conference was held in June in Brigton, East Sussex, UK. Representatives of Cochrane UK attended as exhibitors on both days of the conference with the aim of raising awareness and knowledge of evidence-based decision-making and systematic reviews, and to enable understanding and dissemination of Cochrane evidence. It was also an opportunity to test out different engagement methods that we could build upon in the future.

We spent several weeks planning for this conference and learnt a lot before, during, and after the event;we hope that these lessons will improve the impact of our future conferences and the Cochrane community can benefit too.

Before the conference:

• Know who is likely to attend: list out the stakeholders who are likely to be there and think about how to engage with them. The general public will require different strategies than a specialized group –knowing what mix you can expect beforehand can help with your planning!
• Get the right people: Ensure that your stand is continually manned with people that will work well for the audience you are trying to reach. Given the nature of the conference’s focus and the likely attendees, we decided to have a public health registrar, a retired General Practitioner experienced in pubic engagement, and a public health consultant at the stand.
• Get some help: You can contact Cochrane to discuss your event, get clarification on Cochrane event policies, or help with event branding such as special banners, flyers or branded items to give away.
• Print it out: Have handouts of relevant blogshots or perhaps a flyer about your Cochrane group with your web address. At the very least, when attending a conference have your business cards handy! Templates for conference posters, flyers, business cards, and more in your Cochrane group’s colour are here!
• Create a buzz: Find out the conference’s hashtag and let everyone know you are going to be there! Announce it on your website site, social media accounts, and in your newsletter. We created a ‘virtual conference stand’ with all our information on one page of our website to increase our reach beyond the conference.

At the conference:

• Freebies: Everyone loves something free! We found the cochrane UK pens and notepads to be the most popular.

• Sign-up: We had our MailChimp signup on an iPad so people could sign up immediately to our newsletter if they wanted.
• Quiz time: We made a quiz called ‘Are you making good everyday health choices’ for free with Qzzr and had it on an iPad. It was a great way to have attendees engage with Cochrane Evidence and we could share it further through social media.

• Give a presentation: We had a rolling ‘blogshot’ slideshow on a large screen; this drew people to the stand. I also contacted the committee arranging the conference to arrange to give a short presentation to other public health registrars (health professionals training in public health) about how to get involved with Cochrane UK.
• Get digital: Use the conference hashtag and let people know about presentations at your booth. Take lots of pictures and share them on Instagram, twitter, and Facebook. Use photo opportunities with other organizations and their stands (and their hashtag and therefore their followers) to increase your reach to a new audience.

After the conference:

• Check your metrics: How many retweets? How many newsletter signups? Check your numbers to see if you made an impact. Our 19 twitter posts got retweeted 31 times and lead to 29,083 twitter impressions!
• Write about your experience: Share your experience in a blog post on your Cochrane website or as a guest writer on a related topic to the blog. Including your experiences at conferences and related metrics are also great to include in Annual Reports to your funders.

So much work goes into writing and disseminating Cochrane Reviews. It’s very rewarding attending a conference and sharing Cochrane evidence directly with the people that use it or are affected by it. We leanrt a lot planning for and attending this conference, and hope these tips will help you make the most of your next conference too!

David Roberts

Public health registrar

Cochrane UK

The Cochrane Editorial Unit (CEU) has been undertaking pre-publication screening of Cochrane Reviews since 2013. In that time a team of editors from the CEU has assessed hundreds of submissions, and has not only identified areas for improvement within individual reviews, but also extracted and gathered data to help improve production practices across Cochrane Reviews. In the interests of making this information widely available as a resource for Cochrane contributors, Cochrane editors Newton Opiyo and Toby Lasserson have begun compiling a series of ‘Screening Notes’, which will publish periodically here on the Cochrane Blog.  A PDF of this blog post is available here.

Context
Since pre-publication screening of reviews began in the CEU, we have seen an increase in the number of reviews using GRADE and presenting Summary of Findings tables.¹ However, we continue to see variation in the use of GRADE throughout reviews. This could reflect differences in the planning of GRADE in review protocols. One of the things that we have recognized is the value that GRADE brings to the review process by providing a transparent framework for interpreting results. With this in mind we recently investigated the extent to which GRADE is specified in the protocol as part of the intended methods for the review. We audited two cohorts of review protocols to see if there has been any change in the way that this method is planned in Cochrane Reviews. We were also interested in identifying attributes of good practice in intended use of GRADE and SoF tables in Cochrane Reviews.

Sample & standards
We used two cohorts of protocols published in the Cochrane Library in August 2013 and August 2015. We selected these timepoints to coincide with pre-screening and two years after its introduction. We devised a checklist comprising eight items that we considered relevant for planning GRADE and SoF tables in protocols of Cochrane Reviews:

1. Reference to GRADE as a method for assessing quality of evidence
2. Description of GRADE considerations for assessing quality of evidence (e.g. risk of bias or study limitations, directness, consistency, precision of results, and publication bias)
3. Description of GRADE levels of evidence (high, moderate, low, very low)
4. Description of the methods for preparing SoF tables (e.g. use of GRADEpro software)
5. Consideration for comparisons to be covered in SoF tables
6. Consideration for outcomes to be presented in SoF tables
7. Description of the number of reviewers to be involved in GRADE assessments
8. Description of GRADE and SoF tables in an appropriate section in the protocol

A full description of the methods we used can be found in the full audit report²

Implications of findings
The headline results make for good reading. Importantly, we found a substantial improvement in the number of protocols describing the intention to use GRADE, from 35% to 91%. We found improved reporting of most of the aspects we assessed. A full description of the results can be found in the full audit report².

We also identified key areas for improving the planning of GRADE and SoF tables in protocols of Cochrane reviews. We present four protocols identified from the audit which illustrate learning points for authors and editors to use in their practices.   

Example 1: Total pancreatectomy and islet autotransplantation for chronic pancreatitis (Vallance 2015, CD011799)

“We will create a 'Summary of findings' table including the following outcomes – mortality, health-related quality of life and major complications. We will use the five GRADE considerations (study limitations, consistency of effect, imprecision, indirectness and publication bias) to assess the quality of a body of evidence as it relates to the studies that contribute data to the meta-analyses for the prespecified outcomes. We will use the methods and recommendations described in Section 8.5 and Chapter 12 of the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2011) and using GRADEpro software. We will justify all decisions to down – or upgrade the quality of studies using footnotes and make comments to aid the reader's understanding of the review where necessary. We will consider whether there is any additional outcome information that we were unable to incorporate into the meta-analyses, note this in the comments and state if it supports or contradicts the information from the meta-analyses.”

This first example provides a fair overview of the key aspects of planning GRADE and SoF tables in protocols: reference to GRADE methods, description of GRADE criteria for downgrading quality of evidence, methods for preparing SoF tables and outcomes to be included in the SoF table. We thought that the last sentence in particular showed that the authors were thinking beyond the results of their meta-analysis. It will be useful for readers to be aware of key information about the review findings that cannot be included in the analysis or that provides a more complete summary of the body of evidence. 

Example 2: Virtual reality simulation for reducing pain in children (Lambert 2013, CD010686)

“Results of the meta-analysis will be presented for the main comparisons of the review, the primary outcome child pain and the following secondary outcomes: child satisfaction with virtual reality simulation, child pain-related distress and parent anxiety, as outlined in the section on Types of outcome measures. For each assumed risk cited in the table(s), we will provide a source and rationale, and the GRADE system will be used to rank the quality of the evidence using GRADEprofiler (GRADEpro) software (Schünemann 2011). If meta-analysis is not possible, we will present results in a narrative ‘Summary of findings’ table format (drawing on Chan 2011 as an example).”

In this second example the authors mention plans to prepare SoF tables for the main comparisons of the review. This is particularly important where multiple comparisons are to be addressed. Specification of comparisons to be covered in SoF tables remains an area where improvement is needed. Thirty six protocols included in our audit planned multiple comparisons, but only seven of them mentioned the comparisons to be covered in the SoF tables. Selective reporting of results for a subset of review comparisons may introduce bias, for example, if reviewers only present comparisons that show treatments to be effective or less harmful, or for comparisons where there is most amount of data available. For broad review questions the full extent of relevant comparisons may only become apparent after studies are identified, and may vary depending on the analysis undertaken. Prioritizing comparisons at the protocol stage will help focus the review on addressing questions that users are likely to value most.

Another aspect of good practice in this example is that the authors provide information about the source of assumed risk. This provides a clear basis for translating and communicating relative effects into easily understandable absolute effects. By doing this users will find it useful to determine the applicability of the review findings to practice settings. Decision-makers reading the review will find this information helpful in contextualizing the results of the review.

The authors also use the protocol to provide a commitment to narrative summary of the results in the absence of a meta-analysis. This is crucial as omission of narratively synthesized outcomes in SoF tables represents a source of outcome reporting bias. This a recurrent issue in the reviews we screen. We will write more about narrative SoF tables in a separate Screening Note.

Example 3: Concomitant atrial fibrillation surgery for people undergoing cardiac surgery (Huffman 2015, CD011814)

“Two review authors (MDH, KNK) will independently rate the quality for each outcome. We will present a summary of the evidence in a 'Summary of findings' table, which provides key information about the best estimate of the magnitude of the effect, in relative terms and absolute differences for each relevant comparison of alternative management strategies, numbers of participants and studies addressing each important outcome, and the rating of the overall confidence in effect estimates for each outcome. (…)

In addition, we will establish an appendix 'Checklist to aid consistency and reproducibility of GRADE assessments' to help with standardisation of 'Summary of findings' tables (Meader 2014).”

In this third example the authors outline a process for rating the quality of evidence in duplicate. Given the subjective nature of downgrading decisions we think that this is likely to help embed a structured, consensual approach to rating the quality of evidence.  

Very few protocols in our audit reported on the number of raters involved in assessing the quality of evidence; only four protocols mentioned involvement of at least two reviewers. Duplicate or consensual processes for rating quality of evidence merit greater attention. Studies assessing the consistency of GRADE ratings support the need for more than one rater.^3,4 With this in mind consideration should be given to the use of checklists⁴ in GRADE assessments as planned in this protocol. These checklists should help improve transparency and consistency of GRADE assessments. They may be particularly helpful for those with limited experience using GRADE.

Example 4: Needle size for vaccination procedures in children and adolescents (Beirne 2013, CD010720)

“In the Types of outcome measures section of this protocol we have listed the outcomes (primary and secondary) in terms of perceived order of importance for decision-making and we will include in the 'Summary of findings' tables the first seven outcomes listed. However, as noted in section 11.5.6.2 of the Cochrane Handbook, the importance of an outcome "may only become known after the protocol was written or the analysis was carried out and [review authors] should take appropriate action to include these in the 'Summary of findings' table" (Schünemann 2011a). In the event that during the review process: a) we become aware of an important outcome that we have omitted to include in our protocol or b) we become aware that we have failed to accord sufficient priority to a specific outcome(s) listed in our protocol, then we will include the relevant outcome(s) in the 'Summary of findings' tables. If it is necessary to include outcomes in the 'Summary of findings' tables that were not pre-specified in our protocol, then we will clearly explain the reasons for this in our review, as recommended by Kirkham 2010.”

For this last example we thought that the authors had prepared a reasonably comprehensive overview of the necessary aspects of GRADE and Summary of Findings tables. However, we also noted that whilst the authors had clearly prioritized a certain number of outcomes, they were also mindful of the need to be transparent about changes to outcome selection once the process of data collection was underway. We recognize that anticipating changes to outcomes is not a minimum reporting requirement for all review protocols, but it demonstrates commitment to transparent reporting of changes to methods in the full review.    

Prespecifying outcomes (including methods of measurement, time-points of reporting) helps avoid bias in the choice of outcomes^5,6, and serves as a reminder to review teams to include results for the most relevant outcomes in SoF tables irrespective of the amount and quality of evidence. However, some modifications may be justified; for example, as noted in this protocol ‘the importance of an outcome may only become known after the protocol was written and analysis carried out’. This may be the case for adverse events. Changes to outcomes may also be necessitated by peer reviewers. In these situations authors should explain and appropriately document any change in the specified outcomes.

Summary points
We found a welcome improvement in the planning of key aspects of GRADE and SoF tables in protocols of Cochrane Reviews.  The key learning points from this audit are that:

• There has been an increase in the number of protocols that incorporate plans for using GRADE
• Protocols benefit from defining and prioritizing comparisons for SoF tables
• Duplicate or consensual processes for rating quality of evidence merit greater attention
• Early piloting and independent evaluation of the use of GRADE (once the analysis has been undertaken but before the review is written up) should help improve the implementation of GRADE in full reviews

References

1. Cochrane Editorial Unit. Audit of published new Cochrane Reviews of interventions: 2014 Target 1.2, available from http://editorial-unit.cochrane.org/mecir [date accessed: 10^th February 2016].
    
2. Cochrane Editorial Unit. Audit of planned methods for using GRADE and preparing Summary of Findings tables in protocols of Cochrane Reviews: Target 1.3 for 2015, available from http://editorial-unit.cochrane.org/mecir [date accessed: 24^th February 2016].
    
3. Mustafa RA, Santesso N, Brozek J, Akl EA, Walter SD, Norman G, et al: The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses. J Clin Epidemiol 2013, 66:736–742.
    
4. Meader N, King K, Llewellyn A, Norman G, Brown J, Rodgers M, Moe-Byrne T, Higgins JP, Sowden A, Stewart G. A checklist designed to aid consistency and reproducibility of GRADE assessments: development and pilot validation. Syst Rev. 2014 Jul 24;3:82.
    
5. Smith V, Clarke M, Williamson P, Gargon E. Survey of new 2007 and 2011 Cochrane reviews found 37% of prespecified outcomes not reported. J Clin Epidemiol. 2015 Mar;68(3):237-45.
    
6. Kirkham JJ, Altman DG, Williamson PR. Bias due to changes in specified outcomes during the systematic review process. PLoS One. 2010 Mar 22;5(3):e9810.

This Cochrane Blog post highlights the contributions of Dr. Jimee Hwang, a significant contributor to the work of  Cochrane Infectious Diseases, with an interview. This article is one in the "Meet the Editor" series produced by the Cochrane Infectious Diseases Groups, you can read more in the series here.

Could you describe where you currently work and what you do there?
I am currently a medical epidemiologist at the Centers for Disease Control and Prevention (CDC), seconded to the Malaria Elimination Initiative at the Global Health Group at the University of California San Francisco (UCSF). In my current role, I am the CDC team lead for the President’s Malaria Initiative (PMI) program, led by the US Agency for International Development and co-implemented with CDC, in Ethiopia and the Greater Mekong Sub-region. I support the respective in-country teams to program US government investments and implement malaria control and prevention activities, as well as provide technical assistance in the areas of surveillance, monitoring and evaluation, and operations research. I also participate on several global advisory groups, as well as PMI’s interagency working groups on case management and pre-elimination.

What is a typical day for you?
I spend many days on the road traveling to the Mekong, Ethiopia, or various other locations, both domestic and international. Depending on the time of the year, I can be working on a new annual PMI malaria operations plan for Ethiopia and the Mekong; reviewing other countries’ plans, work plans for implementing partners, and national strategic plans; managing a cooperative agreement; or supporting various operations research projects with protocol development, implementation, analyses, or manuscript generation. Inevitably, there are also many early morning and late night calls to facilitate activities in Africa and Asia.

What prompted you to work in this area?
I was always interested in global health, but imagined that I would be the doctor at the end of the dirt road providing clinical care where it was most needed. My first trip outside of the US was to Kolkata, India as a college student to work in a street clinic called Calcutta Rescue. There, I saw hundreds of children with preventable illnesses like diarrhea and malaria every day, waiting to be seen by Indian doctors. I realized that, for me, the possibility of working in public health and preventing disease to affect the lives of thousands, if not millions, of people was more motivating and exciting than individual patient care.

What are the major challenges that still remain in your field?
The possibility of the world galvanizing to achieve a malaria-free world is astounding, and the opportunity to be a part of this global effort is inspiring. However, this ambitious goal poses many challenges: 1) technically, we need new and improved surveillance and diagnostic tools to tackle the malaria parasite as well as the mosquito vector that transmits malaria; 2) operationally, we need to build well-functioning and managed health systems; and 3) financially, it will take sustained commitment and investment. In addition, effective partner coordination will be essential to realizing this long-term vision.

How did you first hear about Cochrane?
As a busy medical resident, I always looked first to see if there was a Cochrane Review on a topic of interest to guide my clinical practice, as I could trust that the evidence was systematically reviewed.

What is the most rewarding aspect of being involved with Cochrane? 
I find working with an amazing group of dedicated editors and authors that are committed to producing the highest quality reviews very rewarding.

Who (or what) has been the biggest influence on your career to date?
My experience in Kolkata set me on the path of a public health career, and the CDC has built and supported my malaria career. However, the numerous people I have had the pleasure to work with at CDC, The Carter Center, USAID, and UCSF have collectively been the biggest influence. CDC is truly a special and unique institution where I have had the opportunity to be a colleague and friend to an amazing group of dedicated epidemiologists, entomologists, laboratory scientists, and public health advisors. Working in an environment that prioritizes scientific rigor and evidence generation in the context of improving program implementation and impact has really shaped and fostered my malaria career.

Please list three words you would associate with Cochrane.
Trusted, comprehensive, high-quality. 

What do you do in your spare time?
With two young kids, my hobbies now include reading Dr. Seuss, putting puzzles and Legos together, and mastering the monkey bars. I have also re-purposed old interests in woodworking and sewing to build a treehouse or to sew Elsa dresses from Disney’s “Frozen.”  

Thank you for your valuable contributions, Jimee!

Would your Cochrane Group like to tell the community about a beloved contributor? Email mumoquit@cochrane.org