n January, a paper was released that got a fair amount of attention in the parts of the internet that pay attention to the “GMO Debate”, such as it is these days.
For biotech advocates it was a fun, smug moment confirming what we already knew. “The most extreme opponents of GMO foods know the least about science but believe they know the most.” Ta-dah! Dunning-Kruger!!! Nyah nyah nyah nyah nyah nyah!
Those were the conclusions of a paper published in the journal Nature Human Behaviour, a collaboration between researchers at the Leeds School of Business at the University of Colorado Boulder, Washington University in St. Louis, the University of Toronto and the University of Pennsylvania.
“Our findings suggest that changing peoples’ minds first requires them to appreciate what they don’t know,” said study co-author Nicholas Light, a Leeds School of Business Ph.D. candidate. “Without this first step, educational interventions might not work very well to bring people in line with the scientific consensus.”
Social science is hard. Don’t oversell your results.
Before you get on your high horse, the findings weren’t particularly robust. (Remember, as David Dunning of Dunning-Kruger fame reminds us, Dunning-Kruger—a cognitive bias in which people of low ability have illusory superiority and mistakenly assess their cognitive ability as greater than it is—is about US, not THEM. “The first rule of the Dunning-Kruger club is you don’t know you’re a member of the Dunning-Kruger club, Dunning told me in an interview last year. “People miss that.”)
The finding was more a gesture towards a tendency indicated by a clear trend line if you squinted your eyes and ignored some problems with the data. Here we’ll look at the study design and some of the problems associated with the paper before focusing on a secondary finding that was far more straightforward to measure and far more interesting—one that fits in with a range of interesting research on what drives people’s attitudes on GMOs and how it fits into political points of view.
People in the “hard” sciences often dismiss social data, but this paper is a good example of why social scientists actually have the hardest job. In short, people are just a bitch to measure. Unlike plants, soil, cows, atoms, etc, etc, people are touchy SOBs when it comes to assessing them. Just the act of including them in a study can cause their assessment to change, not to mention they can be both independent and highly interactive.
All this is to say, I don’t mean to be overly critical of the authors here. Looking through the paper, they have gone to some good lengths to mitigate problems and view things from several angles. And a major kudos to them for providing ALL DATA and CODES online.
But speaking of data … <here comes the cautionary tale> …
This is Figure 1 showing how one's objective and subjective knowledge can be related to the extremity of one's views. pic.twitter.com/N6j9TStTQm
— Bill Price (@pdiff1) January 17, 2019
These are the data underlying those figures. Life is messy pic.twitter.com/7zhxCYI3ao
— Bill Price (@pdiff1) January 17, 2019
The effect sizes here are a) Small, and b) Vaguely defined indices. Be careful interpreting them. Yes, some trend may be there, but the significance in actuality is questionable to me. Some have described it as “Super Dunning-Kruger”. I think that’s a bit of a stretch.
So, the finding isn’t really robust enough to be all that interesting, the study design and some of the problems with are. Let’s take a look before getting to what they did find that is interesting.
Previous research has looked at the knowledge gap among consumers regarding GE foods, but this is the first study that tried to measure people’s own assessment of their knowledgeability on the issue. In a series of surveys conducted in the United States, Germany, and France, the team attempted to assess consumer opposition and concern towards genetically engineered foods, followed by a self-assessment of consumer knowledge of the subject, followed by a set of fifteen true/false questions to determine scientific literacy.
Five of the questions related genetics and those were further pooled into a subset to assess knowledge relevant to genetic engineering in food. The five questions were:
• Men and women normally have the same number of chromosomes.
• All plants and animals have DNA.
• Humans share a majority of their genes with chimpanzees.
• It is the father’s genes that determine if the baby is a boy or girl.
• Ordinary tomatoes do not have genes, whereas genetically modified tomatoes do.
Problems with the study
The team found the lack of a negative correlation between objective knowledge and extremity of opposition in France and Germany puzzling. When they looked at a larger set of publicly-available data on French and German science literacy and attitudes towards GE foods, they found a pattern similar to the US and consider their results to be false negatives. This is a reasonable fudge when trying to figure out what went wrong, but it points to deeper methodological problems, and it’s a bit odd that they felt they could throw out one data set because it didn’t square with research elsewhere, but forge ahead with the other sets because they did conform with their expected findings.
Elsewhere on Twitter, Brian Lovett, an entomology PhD Candidate, mycologist, and genetic engineer, offered the following observations on the strength of the conclusions versus the underlying data:
The authors first describe the trend of the left graph in figure 1 as trending downward as opposition increases, as this is sort of important for their conclusions. It seems pretty clear the graph levels out at an opposition level of 4 (neutral to GM food). This happens to be the lowest point in the graph, but for some reason, the title “Neutral opponents of genetically modified foods know the least” doesn’t have the same ring to it.
The error bars for points from neutral to most opposed suggest they’re all statistically the same, so it’s not technically wrong to say the most opposed know the least, at best it’s “maybe true”. This wordplay is not surprising, as the paper is full of irresponsible interpretation of statistics. A few gems are “close to significance in France (P = 0.15)”, “marginally significant” and “the non-significant effects in Study 2 are probably false negatives”.
Something becomes marginally significant if it challenges these authors’ narrative, but if P=0.15 and it fits their narrative, that’s “close”. For such broad conclusions, the foundation seems less than stable with such wording.
He also takes a close look at the way the surveys assessed scientific literacy with the 15 questions and people’s confidence in their answers. While the scoring for both correctness and confidence – giving more credit for guessing correctly but being honest about guessing, while penalizing for being confident about wrong answers, seems a reasonable approach, Lovett points out some issues, and more importantly, the range between high results for Objective Knowledge and low results for Objective Knowledge are not all that far apart:
First, the 15 questions were *very* basic. Interestingly, these true or false questions were judged on a scale.
True or false questions, by definition are not on a scale. The first question, is the center of the earth hot, is *definitely true*. However, perhaps to prevent the existential crisis of choosing a side, they allowed participants to hedge their bets for fewer points. So, if you said the earth’s is definitely hot, you get 3 points, probably hot is 2 points, maybe hot is 1 point. If you admit you don’t know, zero points for honesty. If you speculate the center of the earth is cold, how confidently you speculate matters as definitely cold is -3!
Built into their point system is confidence in their scientific knowledge. This seems relevant when you are going to later correlate that “objective” point system with self-reported scientific literacy.
Let’s think about just a couple examples. What if someone does not feel like they know the science? They may report that their GM knowledge is lower, then proceed to report only “probably” answers. If this person gets every question right, they get a score of 30! If they get 9 wrong, they get a -6. What if someone is super confident? They may report high GM knowledge, then proceed to pick only definite answers. If they get every question right they get a 45! If they get 9 wrong, they get an -9. These hypothetical survey participants both answered the same number of questions correctly, but their Objective Knowledge scores could vary up to 15 points based on how willing they were to pick “definitely” over “probably”.
How much does this matter? Well, a lot considering Figure 1. The differences that brought attention to this article are… *squints* less than 10 objective knowledge points from the least opposed to the most opposed. That is basically two wrong questions!
In addition to looking to correlate scientific literacy and intensity of opinion with people’s attitudes on GMOs, the team conducted parallel surveys on attitudes towards climate change and gene therapy. While the results were similar for gene therapy, they did not see the same inverse relationship between science literacy and divergence from the scientific consensus. This is believed to be due to the political polarization of the issue. While negative attitudes among consumers are widespread—in this survey 90.82% of respondents reported some level of opposition to GM foods and 93.01% reported some level of concern — it is linked to a moral intuition rather than political threat. There were no significant differences in extremity of opposition between self-reported liberals, moderates and conservatives. In a previous paper by some of the same researchers their findings suggested that “people find nature and naturalness as sacred and genetically engineered food as a violation of naturalness.” Another paper by the same authors found a sense of ‘disgust’ as a strong predictor of attitudes.
“Absolutist” opponents were more disgust sensitive in general and more disgusted by the consumption of genetically modified food than were non-absolutist opponents or supporters. Furthermore, disgust predicted support for legal restrictions on genetically modified foods, even after controlling for explicit risk–benefit assessments. This research suggests that many opponents are evidence insensitive and will not be influenced by arguments about risks and benefits.
That suggests to me that the battle was lost simply with the adoption of the term “Genetically Modified Organism”, the sense of disgust is built into the name — and the suggestion that there might be tentacles involved.
Feelings of disgust and the violation of naturalness are not intuitions that break along liberal and conservative lines. Climate change attitudes do polarize along political lines with conservatives wary of the government interventions are more prone to find reasons to dismiss climate concerns. In a 2102 paper in Nature Climate Change, Dan Kahan of Yale found that disagreement with climate scientists increased among conservatives as science literacy and numeracy increased. The better educated, better informed they were, the better they got at rationalizing their dissent.
Previous work by Kahan also shows attitudes towards genetically engineered foods to be one of the least politically polarized issues.
This can feel very confusing for people close to and engaged in this issue. In debates it certainly feels politically polarized, and activism on the issue is generally associated with environmental groups that take an interest in agriculture. While it’s true that these groups tend to be aligned in left of center political coalitions, I’d suggest that there isn’t a particular political divide when in comes to activist attitudes towards GE crops. Nearly all environmental groups tend to be left of center, but lots of pro-GMO advocates are environmentalists and skew left as well. The dividing line among active, engaged citizens on genetic engineering is at the center of Charles Mann’s recent book The Wizard and the Prophet.
The book uses the lives of Norman Borlaug, the legendary wheat breeder and father of the Green Revolution, and William Vogt, a key figure in the founding of the modern environmental movement, as a lens to examine the history of environmentalism going back to the middle of the last century. What we find is two distinct ways of being an environmentalist. Wizards are set on finding ways to to do more with less, to decrease our footprint through the application of science and technology. They seek to work around and overcome limits. Prophets are concerned more with finding ways to live within limits. Mann suggests that perhaps the most salient dividing line has to do with attitudes towards the scale of technology. Wizards are looking for the technologies that deliver the greatest leverage for decoupling human activity from environmental impacts. Prophets favor diffuse, bottom up solutions and living in harmony with nature as best we can.
For prophets, the association of genetic engineering with large corporations, industrial commodity crop, and such a command and control approach to breeding bring a deep seated distrust and bias against the technology, even when it is used in Prophet-style projects like Bt Brinjal.
In the uphill battle to counter misinformation and wrong-headed thinking, it’s incumbent to have a sense of which set of values genetic engineering has transgressed before crafting your message. And even then, don’t expect it to have much impact on those who’ve made up their minds.