Either the well was very deep, or she fell very slowly, for she had plenty of time as she went down to look about her and to wonder what was going to happen next.
–Lewis Carroll, Alice in Wonderland
In the current publishing environment, there are a range of options for submitting research papers, among academic publishers that have different levels of rigor in their expectations and enforcement of policies to share data used in writing the papers. Access to the data is crucial, as increasingly large data sets aren’t always clear presented in the traditional publishing environment; only portions of the data may be touched on in the narrative paper body.
But with the Internet, it’s now easier to provide supplemental files or point to data repositories for large collections of underlying results. This permits readers to take an independent look at the actual evidence, disentangled from the interpretations of the researchers involved. Some journals are particularly good about making sure that the data and code on which a study is based are available to readers who wish to explore them to help better understand the claims that are made in a research paper, or who might wish to re-examine or expand on the work.
One particularly admirable policy is that of the PLOS journals. This Open Access publisher states that it is its policy to make the science available to everyone, with “unrestricted access and unrestricted reuse.” This commitment, coupled with their strong and well-defined “Data Availability” policies, should ensure that data from published work is readily available to readers. Its stated policies includes this very clear statement:
PLOS journals require authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception.
PLOS describes numerous ways to make the data available to conform to this policy. Data can be placed in public repositories or provided as supplemental files. Only in rare cases, such as situations that would impinge on patient privacy, would data sharing restrictions be considered valid reasons to withhold certain data types.
But when a paper was published in PLOSOne last summer, and some irregularities and curious conclusions in the text were noticed by readers, scientists began to ask for access to the underlying data which wasn’t available with the paper, counter to PLOS stated policies. And thus began a trip down a rabbit hole that shared many of the perplexing characteristics of Alice’s Wonderland fictional rabbit hole.
On July 2 2015, a team led by Gilles-Eric Séralini published Laboratory Rodent Diets Contain Toxic Levels of Environmental Contaminants: Implications for Regulatory Tests
On PLOSOne. The scientists said that they had tested a variety of animal chows for various potentially contaminating substances. Their analysis claimed to show that pesticides and heavy metals in laboratory animal foods were at levels that would impact the animal’s health, and subsequently undermine the conclusions from any research endeavor. This led to the team to call into question all animal testing data and the subsequent effects the data would have on policy decisions. In fact, this is a pretty bold conclusion:
All these data taken together invalidate the use of historical control data and questions the use of at least 50 rats per group in carcinogenicity studies.
Claiming that all historical control data are invalid is a serious charge.
Immediately, though, it was clear that their conclusions didn’t match the claims all that well. Missing data, and concerns about the statistical evaluations, were noted by readers. So a quest by scientists for access to the underlying data began as early as the day after the paper was posted. Bill Price, a statistician at the University of Idaho, wrote:
Price requested the data for the tested samples, as the summary data provided were insufficient to understand how the data had been treated in the analysis. The “Seralini_group” evaded the issue of data access in a reply:
The European Commission guideline 2002/63/CE is not about biological replicates and statistics, but about sampling methods. Measurements were performed "one-shot" by a validated method, as indicated in the Material and Methods. All the raw data of the study is presented in the supplementary file.
Price went on to explain clearly why the authors’ statement and data files were insufficient, and why non-summarized data are nencessary. Again, the authors evaded the issue of access to the data, which was all that Price requested. This was not satisfactorily resolved publicly in the comment section.
On July 4, two days after publication, I replied to this thread requesting access to the data because I had noted that a test for an herbicide, critical to the paper, appeared to be absent from the published data.
I was also interested in the raw data for pesticides, and was disappointed that it was not yet included. I was curious to see if the omission of glufosinate from Table 1 was just an oversight, or if really was not tested. The paper speaks to not having detected other herbicides, but if you didn't look for one used with the GMOs tested that's hardly a statement to be confident about.
In the paper, the team claimed to have tested the diets to see if they contained glufosinate tolerance traits: “3 were glufosinate tolerant (DAS1507 and T25 maize, MS8RF3 oilseed rape)”. But we don’t have evidence that the presence of the herbicide glufosinate (also known in Europe as phosphinothricin) is among the tested contaminants. It does not appear in Table 1 (Table 1, doi:10.1371/journal.pone.0128429.t001). A quick look at the individual sample data would clarify if it was actually not tested, or if this figure was just missing this item. Séralini and co-authors had written:
As a matter of fact, glyphosate and AMPA, the only herbicide residues detected, were only found in Roundup-tolerant GMO-containing diets, and no herbicides were detected in other samples.
Is this a “matter of fact”? Or did you not test for glufosinate? If you didn’t test it, this statement is a bogus conclusion and should be corrected. If you did, fix the error in the table. And let’s see the data.
But there were other observations in question, too. A tongue-in-cheek re-analysis of the summary data, by Andrew Kniss, a researcher at the University of Wyoming, actually showed a conclusion entirely contrary to the researchers’ claims: the presence of GMO content was associated with reduced hazard.
Further challenging Séralini’s claims, the most “hazardous” feed came from Italy, a country that does not grow GMOs.
There were numerous other data-related issues with this paper. The authors had not been fully disclosing about their conflicts of interest, which corrected by the PLOS editorial staff. And researchers familiar with the type of analysis in this paper were critical of the inappropriate methods and resulting conclusions. Misleading claims about the levels of the hazards and the interactions of contaminants were illustrated by researchers in the field.
PLOS editors unresponsive
Still there has been no release of the data. If it were to be released, some of the questionable matters could be resolved. Or we would be able to build on this data for another more exacting analysis. The stated mission of PLOS includes a commitment to Open Access, which it describes as “Open Access (OA) stands for unrestricted access and unrestricted reuse.” We merely ask for this access.
After the public requests were dismissed by the Séralini research team in the comments section of the paper, on July 9, I requested the assistance of the Data Team at PLOS on this matter. On July 24, the staff wrote me to say that the authors claimed that everything I had asked for was in the supplemental files. I explained that this was not the case, for the same reasons illustrated above. The PLOS publications assistant then bumped the issue to Deputy Editor Dr. Iratxe Puebla. On July 27 I was told this was now under discussion with the Academic Editor, and I would be notified of the outcome. I waited for updates.
By mid-August, having heard nothing, I requested to know the status of this matter. This request was ignored. On September 3, I raised it again. This was not acknowledged. On September 14, I asked again. Finally, I obtained this reply:
I confirm that we are pursuing your concerns about the article. I contacted the Academic Editor in relation to your concerns and the queries about the dataset, and I have approached the authors to request a response to the issues raised and a clarification about the measurements carried out, and the need to supply data for all relevant measurements. I am awaiting a response from the authors and we will establish the most appropriate course of action once we receive their response. (Iratxe Puebla)
Well, that seemed promising. PLOS seemed to understand the issue and were taking it seriously. On September 23 I was told there had been contact with the authors. But as I was still not provided any data, I assumed they had refused to deliver it, contrary to PLOS policies.
Hearing nothing for a month, I followed up. There was a deafening silence. On November 10, I made my final request for an update on this matter. Nothing but silence, again. I offered to include a statement from the PLOS editor in contact with me on the matter with my public comments. None was forthcoming.
European regulators reject Séralini claims because of poor quality of data
So if you have stayed with me as I related my multi-month journey down a rabbit hole, we find ourselves still falling. Not only was there no data to be found, there was also the worrying failure by PLOS and its editorial staff to uphold the publication’s stated data policy. As I had made clear to the PLOS team, this research was being used in an attempt to impact regulatory policy, and it was crucial that the public has access to the data. While this drama was playing out, the Séralini team’s claims that their findings invalidated feeding studies used for regulations were dismissed by independent European regulators partly due to “incomplete reporting of the data”. Petard meet hoist.
Remaining questions for the Séralini team persist. Why won’t you provide the data as promised and required by your publishing journal? Did you or did you not test the other herbicide? If not, will you correct your claims? If so, will you fix the erroneous table? Are there other aspects of the data that are also questionable?
Ironically, not too long ago, the Séralini team had issued this appropriate statement about providing open access to data:
We ask for a free and transparent exchange of scientific findings, mainly when these are related to public health and environmental risks.The public release of these raw data will reveal if significant differences observed between test and control groups in both studies are coherent and if the statistics are of sufficient power in both cases, thereby allowing the design of appropriate follow-up experiments by others, perhaps through a publically discussed and agreed protocol. (Food and Chemical Toxicology, 2013. doi:10.1016/j.fct.2012.11.007)
And yet when Séralini was faced with actually following his own prescriptions for disclosure, he went into hiding. Alas. Transparency for thee, but not for we…?
PLOS under ethical microscope
Questions for PLOS’s editorial staff exist. This very clear case remains unresolved many months after the offending paper appeared. Do you really require that researchers must provide their data, or not?
If not — you are in danger of being perceived as journal in which questionable studies are published and promoted without any requirement that submitters need back up their claims? I would be surprised and saddened to find out this was the case. I hope this can still be resolved appropriately by the editorial team.
I admire the philosophy of the PLOS journals, with Open Access for the work and the availability of the data for mining and re-use. Just last month one of the PLOS founders, Michael Eisen, re-stated this important feature of their mission.
Mining opportunities are a great outcome of open access. And I wish the concept to succeed. But it’s not enough that a paper is open access. You need to insist that the data be provided.
A larger question, though, is what can we do about researchers who make bold claims, but won’t open their work to public review? Where can we turn when the process fails? It’s hard enough these days to distinguish credible journals from predatory pay to play ones. We need to hold the legitimate publishers to their own stated standards. Which means they need to press researchers to live up to their policies. The media needs to know when a story pitched to them is legitimate or not. Government agencies that make policy decisions need to know this too. There are downstream effects of all of this bad behavior.
There should be consequences. But what is the route? Or is it just a deep rabbit hole that lands us in a place where fiction and absurdity rule? That shouldn’t be the outcome. Both the Séralini team and PLOS are derelict in their duties to the public here. But this can be fixed. Just show us the data.
Mary Mangan, Ph.D., received her education in microbiology, immunology, plant cell biology, and mammalian cell, developmental, and molecular biology. She co-founded OpenHelix, a company providing training on open source software associated with the burgeoning genomics arena, over a decade ago. All comments here are her own, and do not represent her company or any other company. You can contact Mary via twitter: @mem_somerville