Last month, the National Institutes of Health released a request for information to get from the scientific community input on the inclusion of preprints and interim research products in grant proposals and reports. Below are responses from ASAPbio and members of the ASBMB’s Public Affairs Advisory Committee, Drs. Rick Page, Matthew Gentry and Susan Forsburg. ASAPbio serves as a focal point for the biological science research community to discuss the role that preprints play in communicating life sciences. The issues surrounding using preprints and interim products is complex, and the responses below represent the opinions of ASAPbio’s leadership as well as the individual PAAC members listed but does not represent the opinion of the ASBMB. We encourage all interested members to respond to the RFI by the Dec. 9, 2016, deadline by going to the submission website.
NIH RFI QUESTIONS
- Types of interim research products your or your organization create/and or host.
ASAPbio: We consider preprints to be complete manuscripts (data, methods, and interpretation) posted on an established, public server typically prior to or during journal-organized peer review.
Individual scientists on the ASAPbio committee, both junior and senior, are posting preprints to disseminate our scientific work prior to journal publication to receive additional feedback and visibility and to accelerate the general pace of scientific communication.
PAAC Members: We, and other scientists like us, produce interim research products in the form of preprints, meeting abstracts, and posters.
- Feedback on what are considered to be interim research products, and how they are used in your field.
ASAPbio: We consider preprints to be manuscripts containing complete disclosures of scientific information: data, interpretation, and methods. While they are typically identical to the version of a manuscript submitted to a journal (for either the initial or revised submissions), preprints could also be used to communicate a broader category of works—for example, negative results, single figures, methods, mathematical/computation models, and data sets. Preprints should always include a description of all methods needed to replicate the work.
While preprints are not yet widely used in biology, their adoption is accelerating. We feel that a wider use of preprints would significantly benefit the scientific process.
PAAC Members: Currently, preprints are the primary form of interim research products used by a subset of the biomedical research community. Although enthusiasm abounds within a vocal minority of the community, clear arguments can be made against the use of preprints for reporting within grant applications or as primary references at the moment. These include:
A) A lack of established community standards for what defines a preprint. At the moment, there is no accepted standard that defines a preprint. How far along must a project be in order to be deemed worthy of dissemination as a preprint? The establishment of such standard, which should be community-driven, will take time. Therefore, we feel it is incumbent upon NIH to properly fund and monitor the development of preprint servers in biology before ascribing significance to these documents within grant applications.
B) A lack of fully established online preprint servers in biology. Unlike established community servers such as arXiv in physics, biology preprint servers are still in an early stage. In fact, a preprint server for chemistry, despite being announced in 2016, has yet to be launched. These new servers need help in being established and supported and we think that NIH would be best served by supporting such servers in their early stages and as they launch.
- Insight on how particular types of interim research products might impact the advancement of science.
ASAPbio: A strong NIH policy concerning preprints will have far-reaching consequences. The development of NIH-endorsed standards for preprints will be critical for their acceptance by the biomedical research community; increased adoption of preprints will benefit both scientists and the research process.
Preprints can accelerate scientific progress in several ways:
A) By quickly communicating new findings to other labs that can build upon the work. In contrast to traditional peer review prior to publication in journals, a process that can sometimes delay the release of findings by several months or even years, the posting of preprints makes the work openly accessible in 1-2 days.
B) By encouraging diverse and more plentiful feedback from peers early in the process of manuscript revision. In contrast to closed journal peer review processes in which 2-3 scientists review a manuscript, the entire scientific community is free to provide feedback and input to authors during the time that a preprint is being revised. As usage of preprints increases, their growing visibility will create the potential for even more commentary.
C) By raising the visibility of research work and promoting collaborations and invitations to meetings. Preprints can connect scientists with similar interests to one another earlier in the research process, presenting more opportunities to work together.
D) By enabling scientists to demonstrate recent progress in the pursuit of research goals, productivity as demonstrated by preparation of a manuscript, and willingness to share findings with colleagues and the public before formal publication. These features can help authors with career advancement, with better informed review of grant applications, and with the establishment of a record of independent work. This is especially important for early career researchers, who are likely to have a limited publication history.
PAAC Members: While preprints offer one possible route for rapid dissemination of research to the community, and are therefore often hailed rightly or wrongly as a means to avoid delays and costs associated with academic publishing, we would again argue that there is no community-accepted standard for ensuring quality of a preprint. Most preprint servers will utilize moderators that serve only to check that a submitted preprint is appropriate for archiving in the correct research category as well as for basic violations such as plagiarism or inappropriate content. These minimal checks are not sufficient to ensure quality within the scientific literature, and may only serve to contribute to the reproducibility crisis in life science, a crisis that has been at the forefront of NIH initiatives in recent years. Thus, it would seem contradictory to us that NIH would at one moment endorse a system whereby there is an obvious lack of checks and balances, while as the same time implementing new policies for increased transparency and rigor in scientific experiments and reporting.
We should also note that although the current peer-review system is by no means perfect, it should not be discarded wholesale through the development of a “subculture” of preprint servers. As our colleague Tricia Serio noted, peer review has flaws that can and should be fixed. While preprints may provide a mechanism for rapid, open distribution of research results, the research community has yet to find an appropriate way to assess or report the quality of science within preprints, and, without this, any preprint system will act only as the antithesis of NIH efforts at reproducibility in life science research.
- Feedback on potential citation standards.
ASAPbio: Proper citation is essential for establishing credit for scientific work; this is inherent to the academic process. If works are not cited in bibliographies, they are less discoverable and their impact is difficult to assess by bibliometric methods.
A preprint represents an initial public communication of scientific findings and should be cited as such. In physics (and increasingly in many biology journals), preprints are cited in a manner similar to journal articles. We believe that this practice is essential for proper attribution of publicly shared ideas and information. To achieve this, preprints should be citable in all grant applications, biosketches, academic assessments, and in the main body of journal article bibliographies. However, we recommend that preprints be identified in bibliographies by a tag that clearly indicates they have yet to be peer reviewed.
In the future, preprints (pre-peer-reviewed manuscripts) and their corresponding journal articles (peer-reviewed manuscripts) should be linked together by tools (such as Crossref’s preprint service) so that their citations can be pooled. This linkage is important for maintaining an accurate and visible record of the evolution of a study. At present, interim versions of the work are easily lost or not recognized.
PAAC Members: It is our opinion that any citation standard is of minor concern compared to the development of standards for metrics of quality. The NIH and US biomedical science is built on a robust peer-review system. Discussions regarding citation standards should be held only after addressing the many other issues concerning preprints. Additionally, we feel that meeting abstracts and posters also do not warrant such citation.
- Insight on the possible need and potential impact of citing interim products on peer review of NIH applications.
ASAPbio: Preprints will enable researchers to publicly demonstrate progress toward goals during a time when their research would otherwise be invisible during the journal review process. For this reason, citing preprints in applications and reports will allow peer reviewers to determine an applicant’s productivity with greater accuracy, particularly the most recent work of the investigator that could be the most relevant to the application under review. For junior faculty who have started their own labs, preprints and other interim research findings could help them to provide more complete evidence of their independence after leaving their postdoctoral lab.
Allowing preprints to be cited in applications and reports would incentivize the creation of these interim research products, which we believe will significantly advance scientific progress as described in question 3.
PAAC Members: At this time, we strongly discourage the immediate use of citing interim research products within grant applications to the NIH. The biomedical research community has not yet deemed the worth of these interim products, and including them in NIH applications would cause both confusion and unnecessary discussions regarding the tool in general rather than the work in question (i.e. a discussion of the utility of interim products rather than the quality of the work in question).
We see the potential for differential impacts within the biomedical research community. Interim research products are traditionally used as a stepping-stone on the path to conventional publication. Thus, reviewers may view interim reports in a negative light. Alternatively, some reviewers may view interim reports as productivity and the impact upon researchers may be slightly positive. These interim research products may enable researchers to more quickly show productivity, but these reports are unable to supplant the preliminary data and proposed research sections within most grant applications.
The impact upon grant reviewers is likely to be unduly high, as the absence of quality metrics for preprints could require reviewers to act as full reviewers not only of the grant application, but also of each preprint. Importantly, interim products have no stated guidelines for reproducibility or rigor. This merely shifts the weight of responsibility of peer review from journal-recruited reviewers to NIH grant reviewers. Reviewers should not be expected to review preliminary manuscripts in addition to the grant proposal. Thus, we strongly urge that these interim products not be included in the grant review process.
- Advice on how NIH reviewers might evaluate citations of interim research products in applications
ASAPbio: We favor the language used in the CIHR Peer Review Manual for Grant Applications, with a recommended addition (bolded):
“Communications, quick-print reports, letters and electronic distribution of pre-prints are important vehicles for disseminating research results. The data presented in [all] such contributions should be treated equally when assessing quality and impact, and reviewers should not regard certain types as “second class” or “grey literature.””
Evaluation of published papers is often strongly influenced by the journals in which they appear. Because the names of preprint servers (unlike the names of journals) bear no assumed relationship to article quality, reviewers should be encouraged to directly evaluate the content of preprints and assess the merit and relationship of the data to grant application. In general, relying more heavily on article content (rather than on journal title) will improve the quality of evaluation of all papers.
PAAC Members: We strongly recommend that NIH reviewers not be tasked with evaluating so-called interim research products. Reviewers already have a very high burden and do not need additional assignments. Until the biomedical research community, including investigators and institutions, comes to a consensus on the methods for reporting assessments of the quality of preprints, we encourage NIH reviewers to treat preprints as evidence of effort in the field, but not as full-fledged publications. This would potentially enable a researcher to more quickly transition to a new field and show initial progress in a grant application without having to wait for publication through traditional channels.
- Any other relevant information.
ASAPbio: To optimize the potential of preprints in the life sciences, these research objects should have maximum visibility and be easily discoverable, abide by data and quality standards, be considered as a “public good” governed by the scientific community, and be open to innovations that will advance the scientific mission.
To achieve these aims, we proposed the creation of a Central Service that could aggregate and distribute (via an API) manuscripts from multiple sources. The Central Service would provide a single place for biologists to look for preprints, a role similar to that served by PubMed for peer-reviewed literature. All manuscripts submitted to preprint servers would ideally be formatted and licensed in a standard way so as to facilitate data mining and other applications. This will provide opportunities for innovation and resources for scientists well beyond the current state of preprints today.
Other forms of interim research products also could be deposited to the Central Service, and we recommend that all software used or created by the Central Service be open source. An international governing body should be created to oversee the operation of the Central Service, as has been implemented for other databases created for the scientific community (e.g. the protein data bank).
As preprints are a public good, we recommend that the cost for the Central Service should be borne by a consortium of funding agencies, at least for the first five years of operation until the establishment of a sustainable business model. The benefits will be large and the cost minuscule compared with publication charges being paid by funding agencies. We encourage the NIH to participate in such a consortium and provide funding for a Central Service and its governance.
We believe that the NIH should develop clear policies on preprints/interim research products and encourage their use in biosketches, applications, reports, and post-submissions materials. These steps will incentivize universities to develop policies regarding the use and evaluation of preprints in academic appointments and promotions and in the fulfillment of criteria for awards of graduate degrees. Preprints could be used in hiring, promotion, and tenure of faculty. Additionally, preprints could fulfill PhD graduate criteria or demonstrate productivity for postdoc fellowships and job applications. Taken together, preprints could promote career transitions and reduce time to independence, a topic of great concern to the NIH and the scientific community in this era in which the increasing age of newly independent biomedical scientists is a matter of national concern.
PAAC Members: We urge the National Institutes of Health to proceed with great caution in opening the door to the use of preprints and interim research products. NIH reviewers already face tremendous workloads imposed by increasing numbers of proposals submitted to NIH. Concurrently, additional documents addressing data sharing and rigor and reproducibility have increased the amount of material that must be reviewed for each proposal. Preprints and interim research products remain in their infancy, and the quality of these products, not subjected to traditional peer review, is unproven. Thus the onus of fully assessing the validity and quality of interim research products will fall to the NIH reviewer, further increasing the burden on an already stressed system. In considering allowing the use of preprints and interim research products, we strongly encourage NIH to consider the following questions:
How does the use of preprints, meeting abstracts, posters, and other interim research projects align with the recent NIH goals to strengthen rigor and reproducibility?
How would NIH allow the use of preprints and interim research products without unduly increasing the burden upon proposal reviewers?
What metrics for quality or what guidelines and reviewer training would the NIH implement that would enable grant application reviewers to appropriately evaluate preprints and interim research products?
While expressing our concern for rushing too quickly into allowing citation of interim research products in grant applications to the NIH, we do not question the goal of the #ASAPbio movement to ensure that publicly funded scientific advances are shared freely and without delay. In fact, we support this goal and laud #ASAPbio for tackling the $9.4 billion per year industry that, at times, capitalizes on the intense competition within biomedical research and the pervasive publish-or-perish mentality. Instead, our concern lies in the ability for the community and the public to gauge and assess quality. Though peer-review is by no means without flaw, it is a system with an established mechanism for assessment. In absence of community-defined quality metrics or processes for assessment, great care must be taken.
In closing, while preprints may develop into an avenue by which scientists within the biomedical research community can more rapidly disseminate publicly-funded research, we recommend that the NIH wait to allow the use of preprints and interim research products and give the biomedical research community more time to arrive at a consensus for implementing quality and assessment standards.