Abel Brodeur (University of Ottawa, I4R)

Alex Brodeur
Université d'Ottawa
Institute for Replication

Coauthors: Mikola, D., Aparicio, J. P., Cook, N., Fiala, L., Abdul Baki, G. & et al.

Abstract

The use of causal identification methods has increased rapidly in recent years, leading to a believed credibility revolution. This paper documents reproducibility and robustness rates by method and sub-field for a sample of 248 published studies, coordinating over one thousand researchers who performed over 15,000 reanalyses (robustness checks). About 91 % of teams reported the original papers to be computationally reproducible from the provided replication materials. Still, approximately 55 % of teams identified some form of research inconsistency (i.e. coding errors, discrepancies). Our teams of researchers found about 70 % of reanalyses remained statistically significant and having the same sign as the original paper. We further classified original papers’ into: (1) their respective sub-fields (e.g., JEL codes) and (2) their design-based research method (i.e., identification strategy). Using a many-analysts approach, we find that results from experimental methods are more robust relative to observational methods. For the sub-fields, we find that development economics and labor economics appear to be more robust than economic history, while political science sub-fields have similar robustness rates. Our results by field and method roughly align with the priors of 250 researchers which were solicited prior to our analysis.