Tracey Perez Koehlmoos is Director of the Health Services Administration Division and Associate Professor of Preventive Medicine & Biostatistics at the US Uniformed Services University of the Health Sciences. She is also a Cochrane Review author, member of the Cochrane Library Oversight Commitee, and contributor to the work of the Campbell and Cochrane Equity Methods Group. Here she talks about her experience of publishing an ‘empty’ review – and the impact of doing so.

It was not the review we had planned. When the team and I first travelled to Geneva in 2007 for training at WHO in the conduct of systematic reviews, we wanted to set the world on fire. In our grant proposal to the Alliance for Health Policy & Systems Research, we had committed to answering pressing questions about the role of the non-state sector in health systems in low- and middle-income countries. We left with a great mentor, Prof. Sandy Oliver, and a burning desire to know: The Effect of Social Franchising on Access to and Quality of Health Services in Low and Middle Income Countries (Koehlmoos 2009).  

After an arduous proposal development process; a screening of the literature that left with me a permanent occupational injury in my mouse-dragging hand; and a tremendous amount of back-and-forth correspondence with the EPPI Centre and the EPOC group; two years later we’d finished the ritualistic dance that meant I was married to this topic until I die or retire. I produced a review that said nothing. OK, not really nothing, but no included studies so just a resounding conclusion that “more research is needed, and this is what it should look like.” Certainly, as scientists when we embark on a journey of discovery, we hate to come away empty handed. Further, all Cochrane discussions on the prevention of empty reviews have stung me for years in a very personal way—as if it was my fault that despite the international hope and brouhaha around the intervention, the literature was immature and I should have known better rather than waste everyone’s time and resources. Further supporters of the intervention were disappointed with my results and held the lack of evidence with disbelief—the problem was with me, I did not dig hard enough or far enough to prove that the good idea worked.

So, what happened? My little empty review took on a secret life of its own. I had clearly stated what needed to happen next in the Implications for Research section of the review, calling for high-quality impact evaluations of the intervention. Then, a number of things happened. The team and I conducted a second systematic review of social franchising with more permissive inclusion criteria and more qualitative analysis that is now published in the EPPI Centre library (Koehlmoos 2011). It provided a platform to discuss the literature that already existed.

Also, we took the lack of rigorous evaluation public by speaking at conferences and in the press through articles in casual places like the ELDIS Health Systems Reporter. The review became included in the Health Systems Evidence database. The lack of evidence for this non-state sector intervention became a topic of discussion at numerous research planning meetings, and was used to generate research funds to test the intervention.

Equipped with an intimate knowledge of the intervention, I became a member of an advisory panel for the largest social franchise in Bangladesh. We supported the completion of a SUPPORT summary on social franchising. The experience of using two different synthesis methods to cover this topic was included in a book on systematic reviews. I watched as primary research studies on social franchising took to the field. As of today in Harzing’s Publish or Perish reminds me that the empty little review now has 39 citations.

I have continued to watch the literature with a standing search of PUBMED so that every Saturday I am updated on publications about social franchising. I have been under pressure to update the review but have not felt that the literature was ready to make the effort worth the resources. In 2011 my life imploded when my husband died in a road traffic crash—and I had to give up my scientific leadership position and research portfolio in Bangladesh as I repatriated unexpectedly to the US. I found myself working for the US Marine Corps—and focusing on the health systems and research needs of Marines and their families. Recently I returned to academia as a professor at the military’s medical school, the Uniformed Services University of the Health Sciences (USU, conveniently located across the street from the NIH). I miss working and living in South Asia and engaging in global development and with international scientists.

Last month in my USU inbox was an unexpected email from an unknown researcher at a US university, “Dear Dr. Koehlmoos, I wanted to update you about my social franchising research…” He went on to say that he has two papers under review. Of course I asked, “Does it work?”, but we will have to wait for the updated systematic review to know for certain. Empty reviews not worth the effort? I strongly disagree.

The opinions expressed in this article are Tracey's own and in no way reflect the opinions of the US Marine Corps, the Uniformed Services University of the Health Sciences, the Department of Defense, or any other agency.