s biotech giant Bayer prepares to spend $10 billion settling thousands of lawsuits alleging its weedkiller Roundup (and its active ingredient glyphosate) causes cancer, we’re forced to address a crucial question: how does an herbicide deemed safe by regulators and scientists the world over become the whipping boy of tort lawyers and environmental groups with an ideological ax to grind?
The answer to that question is complex and difficult to address in a single article, but there are two key factors that helped turn an innocuous chemical into a corporate scandal: the publication of low-quality studies asserting, counter to the expert consensus, that glyphosate poses a serious cancer risk, and gullible media outlets that uncritically reported this research to their audiences. This combination gave lawyers and activists the academic ammunition they needed to pursue litigation and build public support for the false narrative that Monsanto/Bayer ignored evidence of glyphosate’s cancer risk to boost its bottom line.
Using a highly-publicized study from the glyphosate debacle as an example, let’s examine how questionable science slips under the radar of peer review. Although the issues involved will appear to be technical and forbidding, in actuality they can be explained and made accessible. Furthermore, these issues are important for two reasons. First, what is at stake is the availability of a useful agricultural product that farmers value and scientists widely agree is safe and relatively environmentally benign. Second, if there are egregious errors and biases in a paper that has received widespread coverage and has been held up as strong evidence on a high-stakes question, it is important for the public to understand where the errors lie, how the paper could have been published, and how it could have had such an enormous impact.
The Zhang paper
In February 2019, scientists at several universities published a paper claiming that people with heavy exposure to glyphosate had a 41 percent increased risk of non-Hodgkin’s lymphoma (NHL). (Three of the authors had served on the U.S. EPA’s 2016 scientific advisory panel on glyphosate, but dissented from the EPA’s conclusion that glyphosate is not carcinogenic).
Arriving at a time when high-profile tort cases against Monsanto/Bayer were being litigated in the San Francisco Bay area, what I’ll refer to as the “Zhang paper,” after its first author, appeared to deliver a strong scientific analysis supporting an association of exposure to glyphosate and risk of NHL. After all, it was a “meta-analysis” that had the appearance of a high-powered and rigorous assessment of the available human evidence on the question.
Aside from my critiques, published by the GLP and Forbes just days after the Zhang paper went online, and that of another scientist writing at Forbes, the results of the meta-analysis were widely reported at face value. The Guardian, CNN, Reuters Health, Mother Jones, Yahoo! News, the PBS NewsHour, and many other news outlets echoed the paper’s conclusion that there was a “compelling link” between glyphosate exposure and risk of NHL.
In August 2019, NPR’s Pasadena affiliate, KPCC, arranged to have a joint interview of the senior author on the paper, Dr. Lianne Sheppard of the University of Washington, and me. I was looking forward to a spirited debate over her study’s merits, but, unfortunately, two hours before air time Dr. Sheppard pulled out due to “other obligations.”
In February 2020 —a full year after my critique—Dr. Sheppard, wrote an article for Forbes, defending her study. However, rather than addressing any of my criticisms, she implied that I have my own biases and conflicts-of-interest:
“Among scientists who leveled this criticism of our work are Dr. Geoffrey Kabat and Dr. Steven Salzberg in pieces originally published at Forbes. Dr. Kabat’s piece was removed for failure to meet Forbes’ editorial standards, and Dr. Salzberg’s piece referred to some of Dr. Kabat’s analysis without initially acknowledging said retraction (although the piece has since been updated to reflect this). Their arguments echo Bayer’s February 13, 2019 media statement that claimed our paper cherry-picked data.”
But as I explained in a second long piece for the Genetic Literacy Project, this was a rather lame and contorted defense, which failed to explain why my Forbes analysis was removed:
“Dr. Sheppard’s contention that my critique of her study was taken down because I ‘failed to adhere to Forbes’ editorial standards’ is laughable …. What happened, in fact, is that anti-pesticide, anti-GMO, anti-modern agriculture activist Carey Gillam raised a fuss with the editors, and they spinelessly took down the article and severed my connection to Forbes, without any discussion.”
A brass-tacks analysis
In what follows, I’ll give a brass-tacks account of the meta-analysis; the choices and claims made by Zhang et al.; how they justified their choices and claims; and, finally, what the evidence actually says about the link between glyphosate and cancer. In part 2 of this series, I will comment on other aspects of this largely unchallenged paper and discuss what its publication and reception say about the science pertaining to putative environmental risks and how the science is disseminated to the public.
At the outset, we should note that the term “non-Hodgkin’s lymphoma” is a basket term designating a variety of lymphomas (those that are not Hodgkin’s lymphoma), rather than a single entity. Nevertheless, NHL is rare, with an occurrence of about 20 cases per hundred thousand population per year. Also, the incidence of NHL has been flat in the U.S. for the past thirty years, a period which has seen a 15-fold increase in the spraying of glyphosate.
Zhang et al. looked for studies that examined the association of exposure to glyphosate and risk of NHL. They found six studies. These were the large, prospective cohort Agricultural Health Study (AHS) and five case-control studies. The researchers carried out a meta-analysis of the relative risk estimates from these studies.
Meta-analysis is a technique for combining a number of individual studies in order to obtain a more precise and stable estimate of the association you are interested in. It can be thought of as taking a weighted average of the results of the individual studies. The overall risk estimate from a meta-analysis is referred to as the “summary relative risk.”
Meta-analysis was first used to combine the results of small clinical trials in order to obtain a firmer judgment about the effect of a treatment, such as prescribing low-dose aspirin to prevent heart attacks. However, it has become popular to use it to summarize the results of observational (epidemiological) studies, but the latter are often very heterogeneous and lack the protection against bias afforded by randomization.
It is well-recognized that combining the results of studies that are vastly different in quality can lead to spurious results.
Let’s look at the six studies that Zhang et al. combined. The AHS is a prospective study of roughly 54,000 pesticide applicators who were asked about their use of specific pesticides by questionnaire during enrollment in 1993-1997 and again in 1999-2005. Importantly, greater than 80% of the cohort had used glyphosate. Over 20 years of follow-up, 575 cases of NHL were diagnosed in the cohort.
The substantial number of cases and detailed information on glyphosate exposure allowed the researchers to divide the cohort into five exposure levels: no exposure and four increasing levels of exposure to glyphosate. Furthermore, because exposure information was obtained before the development of disease it was not subject to recall bias that is a problem in case-control studies.
In contrast, the five case-control studies were smaller and had more limited information about exposure. The fact that the case-control studies were conducted among the general population means that exposure to any particular occupational or environmental agent is almost always going to be rare. Of the total 2,836 NHL cases in the five case-control studies included in the Zhang et al. analysis, only 136 (or 5%) were exposed to glyphosate. Consequently, the risk estimates were imprecise and uncertain. In addition, there is published evidence from the EPA and other researchers indicating that several of these studies are subject to recall bias and, in some cases, selection bias.
What did the Agricultural Health Study show?
The AHS reported no association between glyphosate exposure and risk of NHL (or with any of over twenty types of cancer). The researchers reported five different analyses, each showing the risk at four exposure levels compared to people with no exposure. The five analyses were for different “latency periods” (latency refers to the time interval between first exposure and diagnosis of NHL), denoted as lag periods of 0, 5, 10, 15, and 20 years.
None of the results of any of the five analyses showed any hint of an increased risk associated with glyphosate exposure. The risk estimate for the highest exposure quartile (Q4) in the five analyses was: 0.87, 0.87, 0.83, 0.94, and 1.12. Essentially, the risk of the highest exposure group in the five analyses was indistinguishable from 1–or no excess risk. Furthermore, there was no evidence of a trend toward increasing risk with increasing exposure level.
The table shows the results of the no-lag and the 20-year lag analyses. All risk estimates in the upper panel are slightly below 1.0 but not statistically significant. And all but one of the risk estimates in the lower panel are slightly greater than 1.0, but, again, not statistically significant. As can be seen, there is no trend toward increasing risk with increasing exposure. In fact, in the lower panel, the lowest exposure group (Q1) has the largest relative risk (1.22), contradicting Zhang’s hypothesis that the highest quartile should show the greatest risk.
Zhang selected the relative risk of 1.12 for the highest exposure group in the 20-year lagged analysis to include in the meta-analysis. Because of the large size of the AHS, this number, even though small in absolute magnitude, made a big difference compared to selecting the 0-lag relative risk of 0.87, which the AHS researchers reported as their main result, or any of the other three risk estimates.
On the whole, with one exception, the risk estimates from the five case-control studies were around 2.0 (the exception showed an odds ratio of 1.0, or no increased risk). When combined with the 1.12 from the AHS—remember that a meta-analysis involves essentially taking a weighted average—the resulting summary relative risk was 1.41 and just barely statistically significant. This was the result that Zhang et al. highlighted in their abstract, declaring that it suggested a “compelling link” between glyphosate exposure and risk of NHL.
Ignoring unhelpful evidence
The first thing to point out is that Zhang et al. simply ignored the four other risk estimates reported in the AHS paper. This is particularly striking because the four other estimates were below 1 and would have resulted in a lower and, in all but one case, a non-statistically significant summary relative risk (as demonstrated in our paper).
How did Zhang et al. justify their selection of the 20-year lagged relative risk of 1.12? The authors stated that, “Our a priori hypothesis is that the highest biologically relevant exposure to GBHs [glyphosate-based herbicides], i.e., higher levels, longer durations, and/or with sufficient lag and latency, will lead to increased risk of NHL in humans.”
But no matter how invested Zhang et al. were in their hypothesis, they were not justified in passing over in silence the four other estimates. In other words, their a priori hypothesis could only be sustained by ignoring estimates which were unhelpful. Furthermore, their total neglect of the other estimates is bizarre in light of this statement from their paper:
“We conducted several sensitivity analyses to evaluate the impact of excluding or including different studies as well as using different RRs/ORs [relative risks/odds ratios] from original studies (Tables 5 and 6).”
This applies to the case-control studies but not to the AHS. An even-handed analysis would have looked at all five risk estimates in order to accurately represent the findings of the AHS.
Much as the authors proclaim their a priori hypothesis, there is a fatal problem with it. As pointed out by the U.S. EPA and by my colleagues and myself, Zhang’s hypothesis is contradicted by what is by far the largest and highest quality study—the AHS. As noted above, in none of the five analyses was there any hint that the highest exposure group had an increased risk of NHL. And there was no evidence of an increasing trend with increasing intensity of exposure.
Because of its size and detailed exposure information, the AHS is the only one of the six studies that can address Zhang’s a priori hypothesis. If the best study on the topic provided no evidence in support of the hypothesis, this ought to have been acknowledged.
It’s not good enough to ignore evidence that contradicts your hypothesis and then go on to combine numbers in such a way as to generate support for the hypothesis. This is simply indulging in circular reasoning. Rather than accepting the 41 percent increase in risk, the researchers should have vetted it to see if it stood up to scrutiny.
What about the focus on the longest latency period, of 20 years? The authors justified this choice with reference to a paper by Weisenberg, which claimed that the median latency periods could be 15-20 years for NHL. However, as we pointed out in our paper, the Weisenberg data were not based on NHL data but, rather, were hypothesized based on an early estimate of the latency period for acute leukemia following exposure to benzene.
A review from the CDC concluded that estimates of latency periods for lymphoma “range from 2 to 10 years.” Not only was Zhang’s choice of 20-year latency not justified by the literature, the authors themselves called it into question later in their paper: “the latency for NHL is uncertain and could be anywhere from 2 years to greater than 15 years.” All the more reason to have examined all five analyses in the AHS!
Thus, neither the claim that an effect of glyphosate exposure should be seen in the highest exposure group, nor the long latency period for NHL posited by Zhang et al. is supported by the peer-reviewed literature.
Rather than focusing on the highest exposure group, the EPA and our group conducted meta-analyses focused on the risk of ever exposure to glyphosate versus no exposure. The two analyses differed somewhat in their selection of estimates from the available studies. Both studies found no association of ever exposure to glyphosate and risk of NHL. While these analyses examined a somewhat different question from that posed by Zhang, unlike Zhang, they made use of all of the data and justified their selection of estimates. If heavy glyphosate exposure was associated with increased risk of NHL, one could reasonably expect to see some indication of an increased risk in those who are ever exposed.
To recapitulate, Zhang et al. carried out a meta-analysis that involved combining studies of very different quality—something that is cautioned against in the Cochrane Handbook—a canonical reference work for the conduct of studies. Their analysis resulted in a small increased risk of NHL, and they concluded that this demonstrated a “compelling link” between glyphosate exposure and risk of NHL.
But in their analysis, they excluded from consideration the bulk of the results from the Agricultural Health Study in order to select the highest risk estimate that, together with the likely biased risk estimates from the case-control studies, resulted in a just-barely statistically significant 41 percent increase in risk. Rather than providing compelling evidence, it is more likely that the Zhang et al. result represents the compounding of a number of biases.
I have only focused here on the most glaring errors in the Zhang paper. In part two of this series, I will examine some of the important questions raised by publication of their paper:
- Are there other errors and indications of bias in the paper?
- How could the paper have passed rigorous peer review?
- How could the glaring bias in the paper have escaped notice?
- How is one to explain that epidemiologists at top universities could publish this paper?
- How do the authors respond to criticism of their work?
- What does the publication of this paper tell us about scholarly standards in the area of environmental epidemiology?
Geoffrey Kabat is a cancer epidemiologist and the author of Getting Risk Right: Understanding the Science of Elusive Health Risks. Find Geoffrey on Twitter @GeoKabat