My team recently encouraged a colleague who is collecting and analyzing data in one of our projects to submit the analysis and results to a journal. His response was “why?” In our first attempt to answer his question, we pointed him to David Evans’ recent post about why people might want to submit short (economics) papers as “letters” to journals that publish letters. David says, “If you believe there is value in expanding the audience, in the quality improvement, and if the citations are useful for you professionally, then it may be worthwhile.”
My colleague argued that his audience is not reading journals, that there are other mechanisms for getting feedback, and that since he is a technical person and not a researcher, he doesn’t need citations. He believes publishing in other ways is equally valued. In other words, he wasn’t convinced.*
This post is my attempt to convince him, and you, that practitioners who are conducting research should publish in journals.
Before giving you my reasons, I think it is important to define what I mean by research. I have in mind a slightly altered version of the US Department for Health and Human Services definition. Mine is: a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to knowledge. My alteration is that I deleted the word “generalizable” before knowledge. To me, a contribution to knowledge means it needs to be more than just information about a specific project useful only to those working on that project. But I’ve found that people can get caught up in what the requirements are for “generalizability” or “externally validity”, so I am avoiding those more loaded terms here. My interest here is when people carefully analyze data or other information to produce new knowledge that can be helpful to others.
So why should practitioners go to the effort of submitting an article to a journal instead of just self-publishing and writing a blog post about it? I see three reasons: because it increases the credibility of the research, it increases the credibility of the practitioner, and it puts the knowledge in the permanent, searchable record. Spoiler alert: I think the third is the most important.
We know that the US Agency for International Development (USAID) and other funders place a premium on evidence from peer-reviewed journals. From my own experience producing evidence products for USAID (see for example the work on STIP), I know that many at USAID do look for whether evidence is from a peer-reviewed journal. In fact, USAID launched its own peer-reviewed journal, Global Health: Science and Practice to make this medium more accessible to both the producers and users of research. Here is a statement from the Gates Foundation highlighting the importance of peer-reviewed publishing.
It is a reasonable question whether the process of publishing in a peer-reviewed journal indeed improves the quality of the research. Many have debated this question (for example here). I personally do not believe that peer review guarantees the quality of the research published in journals, which is why I think that replication is so important. But I firmly believe that the average quality of studies published in peer-reviewed journals is higher than the average quality of studies published elsewhere, even if those published elsewhere are subject to some kind of peer review. You can find empirical evidence to that effect by looking at systematic reviews that do risk-of-bias assessments on included studies that come from various publication sources.
My claim about the relationship between journal publication and quality is a claim of correlation not necessarily causality. Submission to a peer-reviewed journal is an important signal, both that the researcher feels the work is of a standard to be published in a journal and that she is willing to undergo rigorous external peer review. So even if the peer review process itself doesn’t improve the quality (although more often than not it does), the work that was done to get the manuscript journal-ready probably did.
The USAID Bureau for Global Health commissioned this guide on peer-reviewed publishing, which encourages its own staff to publish and states, “One way to show and develop technical leadership is by publishing, especially in peer-reviewed journals.”
Many of you who have made it this far in the post are asking (perhaps emphatically) “if USAID cares so much about journal publication, why do they refuse to pay for it??” Good question. Very good question. It is definitely true that many USAID projects—even some with research as a primary component—do not support journal publication. I think the concepts of time inconsistency and public goods help us to understand this, but I’ll leave that discussion for another post. What I’ll say here is, we need to keep gently encouraging USAID to let their projects contribute credible research to the knowledge base upon which they themselves rely.
To make my third point, about the permanence of journal articles, I’ll start with an anecdote. A few weeks ago, we celebrated the milestone of having more than 10 posts in our new R&E Search for Evidence blog. As I opened the web page to post number 11 and then scrolled backward through the earlier posts, I saw that the post number 1 had disappeared! In turns out the blog was initially programmed to include only 10 posts at a time. We have fixed this, but it drove home the point that blog posts can be ephemeral.
I put the credibility reasons for publishing in journals first, because I think they appeal more to individual incentives. But if the mission of our work is ultimately social impact, then the third reason is the most important. If the question is salient enough to do research, and the research findings are credible enough to disseminate publicly, then shouldn’t we make sure they enter the permanent, searchable, public record?
I will concede one point to my colleague: citations are not such a big deal to technical specialists. Citations reflect whether other researchers are using your research when they write journal articles. If the objective of your research is to inform policy or practice directly, a journal article summarized in a blog post or policy brief may have a big impact without garnering a lot of citations. I do know that people reviewing CVs pay some attention to the quality of journals in which candidates publish, but I have never heard of a case where someone reviewing CVs for a technical position has looked up the number of citations that the candidate has.
Not all practitioners conduct research, so I am not arguing that all practitioners should be publishing in journals. But there are many times when technical specialists do conduct research as part of their project work. Ideally, that would happen more often than it already does, as the need to learn is infinite and our projects are a better source of data than we realize. It is true that publishing in journals is not easy and takes time (another topic for another post) so it is not always feasible. But I think practitioners can and should be publishing in journals more than they currently do.
*In fairness to David, he was not making a case for publishing overall, just for publishing letters, so we misapplied his post a bit in trying to make the initial case to our colleague.
Photo credit: Jessica Scranton/FHI 360