Capacity building survey and questionnaire
In brief, our aims are to proactively identify MEs who struggle to keep up to date with changes, objectives and best practise recommendations, and to address the disparity of MEs who are unable to attend meetings which promote capacity building and networking opportunities.
We recognised this is a multifaceted issue as the Exec addressed this in August 2015 when we surveyed MEs/AMEs on their attendance to past colloquia. We proposed a collection of ideas and presented them to the then Cochrane Steering Group. The purpose of that proposal was to request support to the infrastructure of the organization in order to implement the Cochrane Strategy for 2020 effectively and globally.
In great measure, achievement of Goal 1, Goal 4, and partially Goal 3, rely on the capacity of Cochrane Review Groups (CRGs) to improve their performance, connect with their stakeholders and work with authors to implement strategic actions. Most of these activities will rely on editorial staff, mainly Managing Editors (MEs), to carry them out. MEs will play a vital role in:
- Implementing best practices within the Groups, and
- Fostering communication with authors to implement policies and standard methods and methods development.
Capacity building of MEs is covered and planned only for newcomers through ME Support. However, for those MEs who have been in post for longer than a year, capacity building is available in opportunistic ways, for example at Colloquia, or targeted requests for training by ME Support. This situation has been successful in the past to provide training, based on the self-directed needs of the groups. However, it might fall short for implementing the ambitious objectives of the Strategy 2020 as it relies on MEs identifying their own training needs, being aware of the gaps in their knowledge, and having the opportunity to attend events where their needs are addressed.
This proposal aimed to identify and resolve some of these challenges by providing a strategy to support ME participation in capacity building opportunities, which in turn, will enable MEs to realise the Strategy 2020 objectives.
See full proposal, summary of recommendations and survey questions put to MEs/AMEs in August 2015 here.
MEs/AMEs’ survey
This work lead to a survey conducted by the Exec. One-to-one interviews were conducted between Managing Editors/Assistant Managing Editors (MEs/AMEs) and members of the ME Exec between April and July, 2017. This informal and candid survey was undertaken from a desire for the MEs’ Exec to connect with its constituents and provide an opportunity for MEs/AMEs to discuss any issues related to their work. Of the 77 MEs/AMEs contacted, 64 (83%) participated in a one-to-one conversation. Of the 13 (17%) MEs/AMEs who did not participate: 6 declined the invitation, 3 were on long leave and 4 had recently vacated their post. MEs/AMEs were contacted and the following questions discussed.
- What’s the best thing about your job?
- Finish the sentence, "In my job, I wish I could…"
- What would make your job easier or more efficient?
- Is there anything you are uncertain about?
- What concerns keep you awake at night?
- What change have you made in your editorial system or management of reviews that has had a substantial impact on the quality or timeliness of your work?
- Are there any ways you would like to get more involved in the ME community?
- Would you be interested in attending a small one day ME orientated event held nationally or regionally?
- Do you think your current role allows for career progression?
- What are you doing about continuing profe3ssional development/training as an ME?
- Is there anything else you’d like to add?
Results have been summarized into the following categories.
- Good practices or ideas on editorial management
- Feasibility of a 1-day training for MEs
- Career progression / professional training opportunities
- Concerns
- Highlights
- Solutions for better efficiency
- Additional comments
We conclude this survey with identified action points for consideration.
1. Good practices or ideas on editorial management
Many MEs highlighted the importance of inter- and intra-group communication; noting that employing a trained systematic reviewer in a group can improve the quality of reviews and noting the benefits of resource sharing between groups of Editors and peer reviewers.
Groups have felt empowered through rejecting proposals for new titles, submissions of poor quality, or where there have been unacceptable delays in completion. One group has successfully created and used a memorandum of understanding with authors to hold them to account. Tools and process to help with scoping and prioritisation have been welcomed.
Innovative ideas relating to editorial processes were shared, some of which included:
- requesting more information during title registration;
- requiring a draft Summary of Findings table in a protocol;
- a staged approach to submission that requires authors to submit sections of a review in sequence in order to progress;
- inviting possible contributors to peer referee in the first instance;
- using the RCT classifier to determine percentage of RCTs in a given search;
- or combining sign off and copy editing to speed up the time to publication.
2. Feasibility of a 1-day training meeting
There was unanimous agreement that MEs would welcome a national 1-day meeting, outside of Colloquia. MEs felt it important to have space to discuss concerns and share ideas without fear of criticism. Offers from CRGs to host the event came from the UK (Bath and Liverpool) and Canada - and the ME Exec is working to develop these events. There was also a request to host an event in continental Europe, with a focus on non-native English speakers.
Many respondents said they would welcome training on a variety of topics, which included:
- statistical issues;
- how to spot mistakes;
- new review formats;
- including non-randomised studies; and
- copy editing.
The list of suggested topics will be passed on to the Cochrane training team. MEs recognized a need for rewarding/motivating Editor training – suggesting perhaps CME credits could be awarded.
3. Career progression/professional training opportunities
When asked if they felt there was a path to career progression from their ME role, opinions were almost equally divided.
Many MEs felt their role was not transferable outside of Cochrane and any positions that were advertised were short-term. They felt that continued training and development as being essential in order to keep their skills up to date. However, they felt that available time and training opportunities were limited.
While some have taken up membership with the International Society of Technical and Managing Editors (ISTME), something which MEs are able to do free of cost through Wiley, many find the information overwhelming and would welcome summarized information or pointers to items which may be relevant.
Conversely, a similar number of MEs believed that there are opportunities for career progression. The ability to network with so many other people from around the world and participating in training and meeting events, taking subscription with ISTME and joining Cochrane committees were seen as opportunities to further their careers.
4. Concerns
Many respondents discussed the stress they feel in their role; through the volume of work, the necessity to keep up to date with methods and being under constant scrutiny with respect to review quality and processing time. MEs feel personally responsible for delays to work with very little control over the volume, and increasing demands of the ME role with unmatched levels of support. There was general anxiety about being watched by the CEU but also worry about reviews being wrong, e.g. as fraudulent trials are always a possibility – and this is something we are not well equipped to deal with.
Some respondents felt the ME role had changed over the years from being Editors to Administrators, and that there has been a reduction in ME autonomy and involvement in decision making. There is a desire for greater transparency in Cochrane-wide decision making and consultation with the ME community. Many noted the recent spike in MEs leaving their role and were curious about colleagues’ reasons for leaving. MEs are concerned about how the organisation can ensure quality products and the sustainability of CRGs as experienced MEs move on and that perhaps exit interviews should be conducted to investigate why MEs leave their role. Concern was raised that Cochrane ought to do more to ensure MEs are employed under appropriate contracts and offer means to discuss policies and procedures without fear of recrimination.
Several respondents mentioned that our relationship with authors is changing and that it no longer feels very "Cochrane Collaboration", and more like a traditional journal. There was also anxiety about the rate and breadth of changes being made, to the extent that groups are unable to cope.
The majority of MEs felt uncertain about the Sustainability Review and/or Network Implementation plan and the impact it will have on them. A continued lack of communication regarding structure and function changes was highlighted. There was a sense that each new Cochrane meeting brings new structure and function proposals and policy developments yet it’s not transparent what happened between meetings, or what has changed. With nothing having changed in three years, MEs are receiving mixed messages but no real direction.
Funding was often highlighted as an ongoing concern, with uncertain funding causing stress and having to juggle finances to keep things afloat.
Regarding technology, MEs were concerned that; the needs of authors in LMICs may not have been fully considered by Cochrane, for example, in the adoption of new technologies such as RevMan Web or Covidence that appear to exclude those without the relevant resources, and there are many (>100) items on the UxG wish list for Archie and RevMan which would assist MEs in their role. In particular, while enthusiasm for workflows was highlighted, some feel it is not fit for purpose and some colleagues are not using it.
Ensuring reviews are MECIR compliant has fallen to the groups rather than authors.
5. Highlights
Many MEs highlighted the variety in the work, autonomy to structure their day and intellectual stimulation as positive aspects of their job. Having a role in supporting authors and working as part of a review team were also mentioned as enjoyable tasks, with the overall benefit of helping authors and affecting the health of citizens globally as a broader benefit of the role.
"Cochrane is an amazing organization and it feels good to be part of it and what it is trying to achieve."
"I feel like in my own little way I’m making a difference, and doing something ethical and positive."
6. Solutions for better efficiency
Many MEs stressed the need for better tools to do their job, namely editorial management and technical software. Examples of being able to automatically collate peer referee’s comments, better communication between Archie and RevMan, and improved usability and performance of RevMan (which can be very slow with review that include more than 15 studies).
A variety of support solutions were suggested, all of which equated to more staff, either within the editorial bases or centrally. Many queries that come in to CRGs are generic in nature (e.g. technical questions on how to use RevMan or Archie) and could be handled by someone with less expertise than MEs. Other centralised expertise could include experienced systematic reviewers to help authors complete their review (e.g. SoF tables), Editor, peer review, methodological support, or stats support.
Communication, both between CRGs and the CEU and amongst CRG staff was identified as an area for improvement. Groups struggle with getting Editors, referees, statisticians to adhere to deadlines of their tasks and returning quality work. There was a suggestion that the CEAD could send occasional emails to ask groups when they may be nearing publication as an easy way to engage with CRGs and help open the channels of communication.
Larger, more consistent funding was also identified as something which would help CRGs function better.
7. Additional comments
MEs appreciated the opportunity to talk, with some recommending this exercise be repeated and that the idea be shared with other CRG staff (ISs, Co-Eds) within Cochrane as it ensures the breadth of opinion is captured, as opposed to the usual voices.
MEs were keen that the output of this survey was a set of recommendations with specified times for response and follow up on actionable items.
One ME raised the issue of the lack of recognition by a host institution of the difference in the ME role and Co-Ed roles as the duties of the role overlap so much. This ME is now a Co-Ed but the host institution doesn’t recognise this in promotion terms, and hence pay.
Some MEs would like the opportunity to author reviews, although this would need to be outside of their own groups, so as not to be seen as self-publishing. Not only would this generate a public record of their skills and abilities to use for promotions, it would also build on their CV, allow them to keep up-do-date on methods, add good-quality reviews to the Library, work with other Cochrane staff across review groups, build on their professional network within the organisation, and would also be a good opportunity for sharing editorial practice.
Actions arising
| Whom | Action | When |
|---|---|---|
| MEs Exec | Pass on ideas relating to good practices and solutions resulting from this survey to the CEU for consideration. | Nov 2017 |
| Share example MOU (between authors and CRG) with all CRGs | Nov 2017 | |
| Share other ideas with Editorial board, CEU | Nov 2017 | |
| Pass on list of training suggestions to Cochrane Training | Nov 2017 | |
| Organise 1-day national/regional meetings for MEs | Ongoing | |
| Identify ME Exec member or ME at large to digest the ISMTE information and circulate | Nov 2017 | |
| Plan for bi-annual, confidential contact with MEs | May 2019 | |
| Conduct informal exit-interviews with MEs leaving post | When needed | |
| Follow up on Archie and RevMan wish list items | Ongoing | |
| Follow up with CEU and IKMD regarding needed technical improvements and additions | Ongoing | |
| Encourage and/or arrange regional meetings for MEs | Apr 2018 | |
| Pass on the idea of holding regular one-to-one survey/discussion to other Execs | Dec 2017 | |
| Follow up of recommendations and actions for all. | Ongoing | |
| CET - CEAD | Consider proactively asking what CRGs are publishing in the coming months for promotion | |
| The CEU should be the centralised dissemination point of new methodologies & policies that impact authors | ||
| Increase the frequency, and depth, of communication regarding changes to governance and policy | ||
| Consider how each change to structure or policy will be implemented | ||
| CET – HR | Conduct exit-interviews with MEs | |
| Ensure CRG staff are on appropriate contracts | ||
| Succession planning for Editors and CRG staff | ||
| Assist CRG staff with grading of their post or promotion | ||
| CET – IKMD | Consider LMIC authors’ ability to access new resources, e.g. Covidence, RevMan web | |
| Implement or address recommendations made through the wish list for Archie and RevMan, in particular development on workflows | ||
| Urgent need for better editorial management tools and improvements to Archie and RevMan | ||
| CET – Training | Ensure learning modules highlight MERCIR standards to authors | |
| CEU | Consult with the ME community, involving them in changes to policy or procedures | |
| Celebrate the successes of CRGs, acknowledging things which they do well | ||
| CEU – Editorial board | Consider how MEs can be encouraged to author reviews outside of their group? |
SWOT analysis
| Helpful | Harmful | |
| Internal |
Strengths
|
Weaknesses
|
| External |
Opportunities
|
Threats
|
Goal 1: "To produce high-quality, relevant, up-to-date systematic reviews and other synthesized research evidence to inform health decision-making"
Goal 4: "To be a diverse, inclusive and transparent international organisation that effectively harnesses the enthusiasm and skills of our contributors, is guided by our principles, governed accountably, managed efficiently and makes optimal use of its resources"
Goal 3: "To make Cochrane the ‘home of evidence’ to inform health decision-making, build greater recognition of our work, and become the leading advocate for evidence-informed health care".