We have a reproducibility problem. But is it a crisis?
[This is a copy of my blogpost, previously on thespectroscope.com.]
I often bump into biomedical researchers who do not think that there is a problem with reproducibility. They say it’s just an issue of cancer studies or pre-clinical research. As Francis Collins said at a recent conference that I attended, “If you think reproducibility is only a problem in other fields, you need another look at yours.” It is a serious problem, and it is widespread. But is there evidence that it is a crisis?
I am asking if reproducibility is a crisis because this is crucial for guiding our response to the issue. How much more reproducible was science 20 or 30 years ago? I wish the Bayer study was reproducing another one from them for the 90s. If 70% of published research was reproducible 50 years ago, and now it's the inverse, we have a crisis. However, my suspicion is that we have a problem and reproducibility is a bit worse now than in the past due to the intense hypercompetition in biomedical research, coupled with our 350-year-old publishing system.
In a few days, I will publish a follow-up post with thoughts on effective approaches to increasing reproducibility. I am assuming that this is an old problem that requires attention, but not a crisis that necessitates panic. If I am wrong, please correct me.
[UPDATE 4/11/15: Just came across Stephen Heard's excellent post from a few days ago, with his answer to exactly this question.]
[Comment from Arjun Raj]
I personally think it's an old problem that needs attention, and it's questionable how much attention we should pay to it. There's just not a lot in the way of sensible ways to enforce reproducibility. I think that things that work really spread quickly, and things that don't, well, don't. I think one of the main responsibilities for a PI is to work with their trainees on how to spot when something (a paper, or maybe a whole field) smells a bit funny.