Instant feedback on a Cochrane Review: engaging online with a community of practitioners
Atle Fretheim and Sophie Witter are two members of a four-member author team who prepared a Cochrane Review with the Effective Practice and Organisation of Care Group, focusing on the evaluation of strategies to improve health care and health through performance-based financing. Here they discuss their experience of engaging in an online forum with practitioners of the intervention about the structure, findings, and broader context of their work.
Our Cochrane Review ’Paying for performance to improve the delivery of health interventions in low- and middle-income countries’ was published earlier this year. In an editorial in the WHO Bulletin about our review findings, we stated that “no general conclusion can be drawn regarding the likely impact of performance-based financing in low-and middle-income countries”. This sparked a debate among implementers of performance-based financing (PBF) schemes who are members of the PBF Community of Practice (CoP) Google group. Commenters expressed both frustration and concerns about our findings; many of their comments were focused on the topics outlined below.
1. General scepticism towards systematic reviews
Much of the criticism was similar to that which is often presented in discussions about systematic reviews, and especially about reviews on complex system interventions. For example:
“In your aggregating/synthesizing process, you lose a lot of the context and make oversimplification[s]. Your systematic review lumps together things [that are] very different. You miss the systemic outcomes.”
“I believe that while there is a hierarchy in rigor of the evidence, there is also a hierarchy in its relevance. Systematic reviews score less well on this other scale.”
In our responses to these types of comments via the Google group, we acknowledged the need for contextual information, but pointed out that we had tried to find it, in the form of process evaluations and the like, without much success. We challenged the group to help us identify such reports, which no one did.
We also contended that we had been careful not to “lump” study findings, as demonstrated by the fact that we had not conducted any meta-analyses. However, we did acknowledge that some degree of qualitative “lumping” is a logical consequence of conducting a systematic review.
2. Specific comments about our review
An interesting debate arose about our criteria for study inclusion:
“…the Cochrane Review came too early (in fact I was surprised by the number of studies you found rigorous enough)…”
“What is surprising is that studies were admitted in the systematic review as if they satisfied the Cochrane criteria, which they clearly did notand did not claim to either.”
We found this criticism interesting, since we are much more used to being accused of applying inclusion criteria that are too strict, rather than not strict enough. In the PBF review, we did indeed apply quite liberal inclusion criteria. And the critics may be right that some of the studies should have been discarded. The weaker studies may have added more confusion than clarification to the overall body of evidence.
3. The research question is irrelevant since PBF is known to work
Obviously, our review was conducted on the premise that there are uncertainties about whether PBF actually works, but this assumption was also challenged:
“…there is a problem in setting up RCT[s] because of ethical reasons: If you know something works, it is unethical not to apply it to all levels.”
“…in fact, many questions can be answered by mere logic.”
4. The need for dialogue between the research and practice communities
Some of the frustration that was voiced was about the lack of communication between researchers and practitioners:
“…many practitioners are frustrated by the fact that their voice is not heard enough by scientists and impact evaluators. They are really looking for a balanced dialogue.”
With regard to that: We can hope that our engagement with the PBF CoP represents a small step in the right direction!