Research reproducibility rises to the top

The National Institutes of Health and a highly publicized perspective recently focused the attention of the biomedical research community on reproducibility and rigor. The NIH has been working on reproducibility and rigor for several years since the publication of reports from industry leaders claiming most preclinical research was irreproducible. While NIH policies on research reproducibility have been expected for some time, the new sensational paper in PLoS Biology purporting to quantify the money spent on irreproducible research has brought the community’s issues of reproducibility to a wider audience.

Two weeks ago, the NIH released two notices. The first established NIH policy concerning sex as a variable in biomedical research. The second notice provided a broader set of guidelines to improve research reproducibility. Of importance, these guidelines will affect everyone applying for an NIH grant as they will apply to all types of research from basic through clinical.

The NIH guidelines will require researchers to cite only rigorously conducted studies in grant applications, use sufficiently rigorous experimental design, include a justification for sex bias in experimentation and ensure cell lines and other reagents are authenticated. These guidelines will be in effect for all grant applications with a due date of Jan. 25 or later.

Members of the American Society for Biochemistry and Molecular Biology discussed reproducibility issues with NIH officials last week, voicing the society’s concerns about the burden the guidelines will place on scientists and the speed with which the guidelines are being implemented. As the guidelines were recently announced, the officials did not know how the new rules would be implemented or how they would affect the grant application process.

Published on the same day as the NIH notices, a perspective in PLoS Biology discussed the breadth of irreproducibility in the published biomedical literature. The authors, led by Leonard Freedman from the Global Biological Standards Initiative, broadly defined irreproducible research as “…the existence and propagation of one or more errors, flaws, inadequacies, or omissions…that prevent replication of results.”

Applying this definition to the literature, they claim that between 18 and 89 percent of published preclinical research is irreproducible. The median of this range is near 50 percent, and this is the value that the authors assign as research that is not reproducible. They then state that roughly $56 billion is spent in basic and preclinical research each year; therefore, $28 billion (50 percent) is spent on irreproducible research.

The authors acknowledge the limitations of their study in assessing the true level of irreproducible preclinical research. However, the 1:1 extrapolation of irreproducible research to the amount spent on this research is simplistic and misleading. Scientific articles are composed of many experimental findings, and while some of those findings may not be reproducible, the paper may still yield some important, reproducible insights. Thus, only a fraction of the money spent would have been on research that is irreproducible.

This paper received a lot of press because of the $28 billion supposedly spent on irreproducible science. Science, Nature and National Public Radio provided balanced reporting on this paper, while Stefano Bertuzzi of the American Society for Cell Biology pointed out many of the problems with the analysis. In addition to the issues brought up by Bertuzzi and others online, the ASBMB has another concern. The GBSI is a nonprofit organization that receives funding from the American Type Culture Collection’s BioNexus Foundation. The ATCC and some of the GBSI’s partner organizations, such as Promega, offer cellline and other authentication services for a fee and may profit from sensational reporting of the reproducibility problems in research. Thus, any rules or guidelines spurred by this PLoS Biology perspective may provide a financial benefit to companies that partner with the GBSI. This represents a financial conflict-of-interest.

To be clear, the ASBMB supports efforts to improve reproducibility and rigor in research, and we said as much to NIH officials last week. However, the society questions the burden of the new NIH guidance and pace at which the NIH is implementing them. The society also supports careful, unbiased investigations into the extent of irreproducibility of biomedical research. However, to properly safeguard the public investment in research and to understand the depth of irreproducibility in the literature, these investigations must be conducted from a position of neutrality.

For more on this and other science policy issues, follow the ASBMB Policy Blotter.

One thought on “Research reproducibility rises to the top

  1. As usual, the only counter argument the defenders of the Big Money – Big Science paradigm who penned this article are able to make is an ad hominem attack on the authors. As the authors note, a large part of the reason that their data is so fuzzy is that there isn’t even any standardized coherent criteria for replicating studies, and this in turn is in part (but only in part) due to the fact that there are few standards for conducting them or evaluating the data. These apparently ancillary considerations, in fact, are far more damning of the corruption of modern science by money than the fact that the results of so many studies can’t be replicated.

    The only solution to this corruption is to get the government completely out of the science business, both directly and indirectly. The indirect factor in this case is actually the more important one. The study notes that about two thirds of this junk science is being cranked out by the drug companies who are all having to dance to the FDA’s ever-changing, arbitrarily selected tunes. The FDA is both unconstitutional and responsible for the deaths and suffering of countless millions of people who can’t afford to travel out of the country to obtain the drugs or therapies they need. Naturally, this doesn’t matter to the bureaucrats or the bureaucrat scientists who control the regulations and the flow of grant money, since they are all well up in the top 10%ile of Americans by income. So let the rest of us eat cake.

Leave a Reply

Your email address will not be published. Required fields are marked *