Reproducibility in science is not very sexy. Because our scientific culture generally rewards innovation over cautiousness, replicating a study conducted by others will not get a researcher a publication in a high-end journal, a splashy headline in a newspaper, or a large funding grant from the government. In fact, only an estimated 0.15 percent of all published results are direct replications of previous studies. We tend to take published studies at face value, assuming that the data and conclusions presented are accurate. Then we build upon these studies, using them to generate new hypotheses, new experimental designs, and new clinical trials, in a cycle that repeats over and over again.
Unfortunately, in the past few years, many studies have reported that the majority of results within biomedical research cannot be replicated. The Reproducibility Project: Cancer Biology, an open-source replication project established in 2013, aims to examine this problem by attempting to replicate the results of fifty recently published landmark preclinical cancer papers. The project hopes to estimate the extent of the reproducibility problem and identify its contributing factors. With the project’s first experimental results due out this fall, opinions are mixed as to how useful this study will ultimately be. Many scientists fear that the project’s open-source methodology will initiate a witch-hunt that could have negative impacts on the funding and reputations of individual investigators. Despite this apprehension, the majority of scientists agree that scientific reproducibility is lacking across the board.
In 2011, pharmaceutical companies, Bayer and Amgen, reported reproducibility rates of 11 percent and 25 percent, respectively, following systematic testing of published biomedical studies. Furthermore, a 2013 survey published by the MD Anderson Cancer Center in Houston found that over half of their researchers were unable to reproduce the results of a published scientific study (Figure 1).
Figure 1: Examining scientific reproducibility has been the focus of many studies, reports, and letters in recent years.
These groups rarely accused the researchers who had performed the original studies of willfully committing fraud. Rather, they identified irreproducibility as a systemic problem that occurs due to an intense environment that rewards publishing positive data. They also cited insufficient statistical analysis and immense competition among labs as possible contributing factors. No matter the cause, irreproducibility continues to be a detriment to the scientific community.
It is estimated that the United States spends $28 billion dollars each year on preclinical cancer studies that produce results that cannot be repeated. Beyond this possible financial toll, irreproducibility may waste the time of researchers each year. Since published studies are the basis of future studies, hypotheses, and clinical trials, one irreproducible paper is likely to have a ripple effect into other laboratories and companies — a potential waste of years of effort. In fact, according to an analysis published in 2012, a significant contributor to the high failure rate in oncology clinical trials is the poor quality of published preclinical data. Reproducibility is necessary to filter out subtle biases and errors not caught by the original researchers. If a study is reproducible, scientists have more confidence that the broad conclusions are widely applicable.
Reproducibility Project: Cancer Biology
The Reproducibility Project: Cancer Biology was established by the Center for Open Science, a group based at the University of Virginia aimed at promoting transparency in science. The group, in collaboration with contracted laboratories from the Science Exchange, aims to examine the reproducibility problem by systematically repeating the results of fifty pre-clinical cancer studies. According to the study leaders, the Project’s first replication study will be published this fall, with all fifty replication studies published by the end of 2017. The ultimate goal is to produce a large dataset that researchers may use to a) estimate the extent of the problem of irreproducibility and b) reexamine the current research and publishing practices that lead to irreproducibility in cancer biology.
These goals are not new, as previous reports have identified key factors influencing irreproducibility. In fact, in a 2013 open letter to scientists, leaders of the National Institutes of Health identified some systemic problems contributing to irreproducibility including: poor training of researchers in experimental design, increased emphasis on provocative statements over presentation of technical details, and lack of clear reporting of experimental instructions within scientific publications. The pressures of our system lead to studies that are not reliable enough to withstand replication, often because the conclusions are not as broad or simple as claimed.
What makes the Reproducibility Project: Cancer Biology unique? Unlike previous groups, this team is conducting their study in an entirely open manner. All fifty papers under examination are listed by name on the project’s open wiki website for public viewing. Each of the fifty papers will go through the Project workflow (described in Figure 2) to determine whether their key findings, often involving two to three experiments per study, can be replicated. Crucially, this process involves open collaboration with the lab that originally published the paper. Before any experiment even begins, the Project researchers create detailed protocols and publish them for review by the original lab and other relevant scientists. This uniquely transparent process aims to ensure the best possible chance of replication. Despite the project’s best efforts, there is a multitude of concerns surrounding this endeavor.
Concerns regarding Reproducibility Project
These concerns are related to three valuable resources: money, time, and reputation. Thus far, the Reproducibility Project: Cancer has been awarded $1.3 million dollars plus a half million dollars’ worth of donated scientific materials, roughly $26,000 per study, to fund the replication work. In our current funding climate, it is difficult to justify funding purely replicative studies, but if the insights from this paper are able to alleviate the reproducibility problem, perhaps we will be saving money in the long-term. The project has also reportedly expended a significant amount of time of the laboratories that published the fifty cancer papers. Most labs cited 20 – 30 email exchanges with the Reproducibility Project, in addition to up to two weeks of full-time work by a graduate student, to retrieve key procedural details of the paper.
More than being a potential waste of time and resources, a major concern is that if this study reports low reproducibility rates, individual reputations and public support for biomedical research may be damaged. There is reason to believe that these fears are justified. The Reproducibility Project: Psychology, a similar venture that tested one hundred psychology studies, was released at the end of August 2015 and within days, the headlines rolled in: Nature’s “Over half of psychology studies fail reproducibility test” and The Guardian’s “Psychology research: hopeless case or pioneering field?,” among others. The public evaluation of large-scale replication projects could undermine individual, institutional, and overall field reputations quite easily.
Beyond these concerns, many scientists are worried that direct replication by contracted laboratories is not the way to examine the reproducibility problem. Given that the scientists performing the reproducibility studies may not have the same expertise as the scientists that conducted the original studies, most authors will not be surprised if their papers are not replicated. Jeff Settleman, the CEO of Calico Life Sciences and an author of two of the fifty papers, described this issue to Science using the analogy of a recipe: “You can’t give me and Julia Child the same recipe and expect an equally good meal.”
Researchers at the Reproducibility Project: Cancer Biology disagree. They maintain that not only are their replicating laboratories highly specialized, but that if the recipe (in this case, the experimental methodology described in the paper), has enough detail, two different people should be able to produce the same meal.
By all accounts, the leadership of the Reproducibility Project: Cancer Biology is acting in good faith with a clear desire to replicate as many results as possible. It remains to be seen whether this levelheaded attitude will extend to how results are actually reported by the media, or if the study will yield its own set of sensational headlines to the detriment of the cancer research field.
In a country where biomedical research funding has been steadily declining over the past decade, it is natural to wonder whether devoting time, money, and reagents to such large-scale replication studies is worthwhile. There have already been many smaller studies and analyses to examine this problem, and it is unclear whether this approach will reveal any new insights. While there are concerns about how the results of The Reproducibility Project: Cancer Research will be published, there is at least one benefit: all of the researchers are bravely contributing to the discussion about our scientific culture. As Brian Nosek, the founder of the Center for Open Science and the lead author of the Reproducibility Project: Psychology study told The Atlantic, “I really hope that this isn’t a one-off, but a maturing area of research in its own right.” Perhaps reproducibility is getting a little bit sexier.
Steph Guerra is a Ph.D. student studying cancer therapeutics and the co-director of Harvard’s Science in the News. Follow her at Steph_Guerra_