Three days ago, I asked whether irreproducible research is a problem or a crisis. I asked on Twitter, consulted Ivan Oransky of RetractionWatch, and discussed over e-mail with many.
We have a reproducibility problem. But is it a crisis? thespectroscope.com/read/we-have-a… (I am genuinely curious about the answer. Please RT.)
29 people are talking about this
The overwhelming consensus is that it is indeed an old problem rather than a new crisis. Most have heard that Bayer and Amgen could not reproduce 75-90% of published research. This has been covered extensively in the media (Economist, Wall Street Journal, NY Times, Reuters, Boston Globe). But Bayer and Amgen haven't done their reproducibility studies 20 years ago to be able to say that what we have today is significantly different, and there is simply no evidence today that the situation is dramatically worse.
And if this isn't a new crisis but an old problem, that has huge implications for how we address it. There have been some extreme reactions to the reports of the irreproducible research (let's require independent validation of results before publishing, refund grants for retractions, or insitute random audits of labs). Let's calm down and keep in mind that this same inefficient, messy, hard-to-reproduce research enterprise has given us extraordinary breakthroughs throughout the last century and on economic terms gives phenomenal returns on taxpayer funding.
At the same time, we shouldn't just pat ourselves on the back and ignore the problems. Just because the return on the taxpayer investment is good, doesn't mean we shouldn't strive to make it better. I am against heavy new regulations and creating bureaucratic overhead at a huge cost, siphoning away money from new research into audits and validation of old results. I am against efforts that throw extra burdens on the scientist when doing the work and publishing it; the scientist is overwhelmed and stressed enough as is. But there is so much we can do if we acknowledge the problem rather than panic about the non-existent crisis.
If we acknowledge as the Bayer/Amgen reports show and as John Ionnidis argues that Most Published Research Findings Are False, perhaps the best way to deal with it is to make it easier to correct the research rather than obsess with making it perfect in the first place? Our whole company exists to make it easier for scientists to share and discover corrections and optimization of published methods on protocols.io (great post from Mike Loukides on why this is so critical for reproducibility). Here are some of the many other terrific efforts that address the reproducibility problem in biomed while making scientists' lives easier:
- Open Peer Review - if we see the concerns of the reviewers and the responses of the authors, that makes it easier to determine what we can and cannot trust in the published work. F1000Research, BMJ, Publons
- Post-publication reviews/discussion/annotation - it is a foolish to think that peer review can predict whether or not a given work will stand the test of time. Peer review is not designed for that. But as the authors and other researchers extend on the published work and invariably find mistakes and corrections, we need to make it easy to correct! PubMed Commons, PubPeer, F1000Research, protocols.io
- Pre-print servers - one contributing factor to irreproducible research is the pressure to publish in a high impact journal and the desire to do everything possible to see your work finally in print. The pre-print servers relieve so much of this pressure by letting you instantly make your manuscript public. bioRxiv, PeerJ, F1000Research
- Discovery services - Since the nature of science is dynamic and evolving and most publications need corrections and optimizations, it's important to automatically keep the researchers informed when these changes happen. That's why our PubChase users instantly get notified if a paper in their library is being discussed on PubMed/PubPeer or has been retracted. Our protocols.io users are alerted right away if a protocol they use has been annotated. And services such as ReadCube, Mendeley, Papers, Zotero are in a great position to implement as many of such auto-notifications as possible.
The very nature of science research, because it's pushing the boundaries, is that much of what is published is inherently wrong. Just as software released by startups cannot be free of bugs, striving to publish research that is 100% verified and correct would undermine the progress of science. Good startups fix these bugs quickly. Let's make it easier for the scientists to do the same.
------------------------
[UPDATE 4/11/15: Just came across Stephen Heard's excellent post from a few days ago, making a similar argument.]
---------
[Comments below]
-------------
[Arjun Raj]
Hi Lenny, I agree completely. I think a lot of the problem is because of the overemphasis of statistical inference in modern biomedical science. I think the proposed solutions are good ones. I would add another, which is training young scientists to recognize what is real from what is likely bogus. There are some tell-tale signs that we can train people to recognize.
-------------
[D Samuel Schwarzkopf]
Great post! This is pretty much what I've been trying to say. At that UCL event on fixing science last month I in fact mentioned that the "replicaton crisis" is very old as are many of the other problems we are currently worry about so much. I believe scientists should -always- consider how the process of science can be improved just as they should always think about how to improve their experiments - that's what we mean by saying science is self-correcting.
I believe science has come a long way from the early 1900s when it was almost exclusively the domain of well-to-do white men, there was no formal peer review, only basic statistical techniques were available to scrutinise results, and nobody had even heard of "open science." Obviously, we shouldn't pat ourselves on the back. It's a constant process of self-improvement. The suggestions you make all make perfect sense to me.
Sam Schwarzkopf
-------------------
[Stephen Heard]
Interesting perspective - nice point about the cost to progress of insisting that every result be "verified and correct". I recently made some similar arguments here. The "crisis" is indeed very old - Robert Boyle worried about it in the 1600s!
No comments:
Post a Comment