The National Institutes of Health and a highly publicized perspective recently focused the attention of the biomedical research community on reproducibility and rigor. The NIH has been working on reproducibility and rigor for several years since the publication of reports from industry leaders claiming most preclinical research was irreproducible. While NIH policies on research reproducibility have been expected for some time, the new sensational paper in PLoS Biology purporting to quantify the money spent on irreproducible research has brought the community’s issues of reproducibility to a wider audience.
Two weeks ago, the NIH released two notices. The first established NIH policy concerning sex as a variable in biomedical research. The second notice provided a broader set of guidelines to improve research reproducibility. Of importance, these guidelines will affect everyone applying for an NIH grant as they will apply to all types of research from basic through clinical.
The NIH guidelines will require researchers to cite only rigorously conducted studies in grant applications, use sufficiently rigorous experimental design, include a justification for sex bias in experimentation and ensure cell lines and other reagents are authenticated. These guidelines will be in effect for all grant applications with a due date of Jan. 25 or later.
Members of the American Society for Biochemistry and Molecular Biology discussed reproducibility issues with NIH officials last week, voicing the society’s concerns about the burden the guidelines will place on scientists and the speed with which the guidelines are being implemented. As the guidelines were recently announced, the officials did not know how the new rules would be implemented or how they would affect the grant application process.
Published on the same day as the NIH notices, a perspective in PLoS Biology discussed the breadth of irreproducibility in the published biomedical literature. The authors, led by Leonard Freedman from the Global Biological Standards Initiative, broadly defined irreproducible research as “…the existence and propagation of one or more errors, flaws, inadequacies, or omissions…that prevent replication of results.”
Applying this definition to the literature, they claim that between 18 and 89 percent of published preclinical research is irreproducible. The median of this range is near 50 percent, and this is the value that the authors assign as research that is not reproducible. They then state that roughly $56 billion is spent in basic and preclinical research each year; therefore, $28 billion (50 percent) is spent on irreproducible research.
The authors acknowledge the limitations of their study in assessing the true level of irreproducible preclinical research. However, the 1:1 extrapolation of irreproducible research to the amount spent on this research is simplistic and misleading. Scientific articles are composed of many experimental findings, and while some of those findings may not be reproducible, the paper may still yield some important, reproducible insights. Thus, only a fraction of the money spent would have been on research that is irreproducible.
This paper received a lot of press because of the $28 billion supposedly spent on irreproducible science. Science, Nature and National Public Radio provided balanced reporting on this paper, while Stefano Bertuzzi of the American Society for Cell Biology pointed out many of the problems with the analysis. In addition to the issues brought up by Bertuzzi and others online, the ASBMB has another concern. The GBSI is a nonprofit organization that receives funding from the American Type Culture Collection’s BioNexus Foundation. The ATCC and some of the GBSI’s partner organizations, such as Promega, offer cell–line and other authentication services for a fee and may profit from sensational reporting of the reproducibility problems in research. Thus, any rules or guidelines spurred by this PLoS Biology perspective may provide a financial benefit to companies that partner with the GBSI. This represents a financial conflict-of-interest.
To be clear, the ASBMB supports efforts to improve reproducibility and rigor in research, and we said as much to NIH officials last week. However, the society questions the burden of the new NIH guidance and pace at which the NIH is implementing them. The society also supports careful, unbiased investigations into the extent of irreproducibility of biomedical research. However, to properly safeguard the public investment in research and to understand the depth of irreproducibility in the literature, these investigations must be conducted from a position of neutrality.
For more on this and other science policy issues, follow the ASBMB Policy Blotter.