How to conduct a replication study: What not to do

Two weeks ago, on Halloween, I wrote a post about how to conduct a replication study using an approach that emphasizes which tests might be run in order to avoid the perception of a witch hunt. The post is based on my paper with Benjamin D.K. Wood, which I recently presented at the “Reproducibility and Integrity in Scientific Research” workshop at the University of Canterbury. When Ben and I first submitted the paper to Economics E-journal, we received some great referee comments (all of which are public) including requests by an anonymous referee and Andrew Chang to include in the paper a list of what not to do – a list of don’ts.

We spent some time thinking about this request. We realized that what the referees wanted was a list of statistical and econometric no-nos, especially drawing on the most controversial replication studies funded by the International Initiative for Impact Evaluation (3ie) while we were both there. However, our role at 3ie was to be a neutral third party, at least as much as possible, and we didn’t want to abandon that now. At the same time, we did learn a lot of lessons about conducting replication research while at 3ie, and we agreed that some of those lessons would be appropriate don’ts. So we added a checklist of don’ts to the paper that was ultimately published. Here I summarize three of these don’ts.