The GLP is committed to full transparency. Download and review our 2019 Annual Report

When is it time for a scientist to call out peers over questionable research?

| August 30, 2018
retraction
This article or excerpt is included in the GLP’s daily curated selection of ideologically diverse news, opinion and analysis of biotechnology innovation.

Recently, I received an email from a wildlife researcher — I’ll use the pseudonym Scientist A — who wanted to anonymously seek advice on a professional quandary. This researcher believes that two colleagues are presenting data on a controversial wildlife species in a misleading way. The dataset now spans roughly four decades, but when the method for counting the critters changed in the 1990s, the population census leapt by an unprecedented amount. While Scientist A says the switch in survey methodology created the illusion of a spike in the animal’s population growth, the colleagues — whom we’ll refer to as Scientists B and C — portrayed the increase as a biological reality. Scientist A further notes that although their older research reports clearly mentioned the shift to the newer census method, the more recent studies by Scientists B and C haven’t acknowledged it. That includes a new report this year that ignored other published work pointing out the importance of the change in methods. As a result, Scientist A is now pondering whether the right move is to write another commentary and continue discussing the issue publicly in the literature — or demand retraction of recent papers by Scientists B and C. “Is this a case for scientific debate,” Scientist A wondered in the email message, “or a case of misconduct?”

Let’s start with the bad news for Scientist A: It’s impossible for us at Undark to really weigh in on the particulars of this conflict without knowing a great deal more detail from both sides. Scientist A views things one way, and Scientists B and C clearly view it differently — and we are in no position to adjudicate the findings of wildlife population science in this space.

That said, the more general issue confronting Scientist A — suspecting a misstep in the published work of colleagues and being uncertain of how to effectively deal with it — is not that unusual in the world of science. Errors that compromise the research literature, even just unintentional mistakes, continue to be a major problem, after all. And calls for more aggressive policing of peer-reviewed work have been on the rise for years, as have attempts to create safe-spaces for scientists to own up to their own research mistakes.

But at the heart of Scientist A’s lament are some fundamental questions that, sooner or later, every researcher is likely to run into: What’s the best way to address suspected inaccuracies or misleading information in the scientific literature, which is supposed to reflect our best approximations of reality? And how best to engage colleagues when there’s a dispute over data, attribution, methodology, or other aspects of presenting research findings?

There are no easy answers, but setting the record straight — and doing so in a transparent and professional manner — is all the more important when research results can shape policies that affect public health, the environment, or in this case, wildlife conservation and management. To get some perspectives on navigating this sort of predicament, I checked in with a few experts.

“We know that there’s a lot of inaccurate material out there,” said Nicholas Steneck, a science historian at the University of Michigan and consultant on research ethics and integrity. While that doesn’t mean that good science is not being done, he told me, “it certainly slows research, it wastes a heck of a lot of money, and we ought to do more to clean up the literature than we actually do.”

As one case in point, Steneck recalled a postdoctoral researcher who told him that leading investigators in an emerging area of biomedical science had improperly calibrated some equipment. Consequently, most of the early studies were misleading, yet no one was willing to address it. “The leaders did not want to retract their earlier research and the postdoc was not willing to risk a career,” Steneck said. Dozens of the unreliable articles remain in the published literature, and while most experts in the field know that, outsiders may not have a clue.

retracted 8 13 18 2

Calling out others’ possible research blunders can be risky, and reluctance to do it is understandable. Those who wish to take action need to first take stock of how certain they are of their own conclusions, of course. But they also should consider how protected their own job position is, Steneck said. In the case of Scientist A, for instance, if Scientists B and C wield a lot of influence in their field, could criticizing their work hurt Scientist A’s ability to get funding for future projects? If so, that’s a real problem, said Steneck, who suggested that the wildlife researcher seek advice and support from a supervisor or department chair, as well as the scientist’s institution.

“When you do go into those situations,” said Steneck, “it’s best not to do it alone.”

While recent concerns about shoddy science have largely focused on problems in the areas of psychology and biomedicine, the field of ecology isn’t immune to the publish-or-perish pressures that encourage flawed and exaggerated research reports. Ecologists and evolutionary biologists commonly engage in so-called questionable research practices, according to a survey published last week by Hannah Fraser and Fiona Fidler of the University of Melbourne in Australia and Tim Parker of Whitman College in Walla Walla, Washington. Two-thirds of respondents admitted that they had, for instance, cherry-picked data on at least one occasion. (Such questionable research practices generally aren’t considered misconduct, though some people argue otherwise.)

When I briefly described Scientist A’s concerns about the wildlife population dataset to Parker, an ecologist and evolutionary biologist, he thought the situation sounded problematic. Most researchers would agree that studies should always report that a change in measurement methods was implemented at the very time-point when a big change in the measurements was observed, he said. Scientists are supposed to rigorously look at all the evidence bearing upon a question and evaluate it openly, so that others can assess whether their conclusions are right. “If you’re concealing some piece of information that undermines the case that you’re making, then clearly you’re not being transparent,” Parker said — and that, to him, potentially smells of misconduct.

Related article:  Viewpoint: What defines a female athlete? Law professor, former runner's case for why Caster Semenya’s testosterone levels critical in determining if she should compete with women

However, by definition, scientific misconduct means that perpetrators acted intentionally to commit fraud, and that’s tough to prove.

What about pushing for retraction of the studies in question? If a publication is inaccurate — whether due to honest mistake, or misconduct — it should be retracted. Alternatively, if a study with otherwise sound conclusions nonetheless contains erroneous data, the paper should be corrected.

The trouble is, pursuing retractions or corrections is often an uphill struggle. Generally, going through all the “proper channels” for addressing flawed research — from writing letters to authors or journal editors to requesting a retraction — seldom works, said Ivan Oransky of Retraction Watch, a website dedicated to tracking retractions made by scientific journals.

While Retraction Watch doesn’t give advice on troubleshooting specific cases of questionable research, Oransky shared a few broad observations. “It may sound strange coming from someone who’s cofounder of Retraction Watch, but retractions are not always the best solution,” he said. If the goal is to clean up the scientific literature, demanding a retraction may be counterproductive because it instantly puts everyone “on edge,” he said. “If you immediately request retraction, because it’s got a stigma, because it’s got an association with misconduct and fraud, someone’s just going to get defensive.”

So where does that leave Scientist A? In cases like this, perhaps the best solution for dealing with inaccurate findings, Steneck said, is to “collect evidence and publish peer-reviewed articles that call the prior research into question.” He pointed to the example of Nathan Myhrvold, the wealthy former chief technologist at Microsoft with a physics Ph.D., who in 2016 disputed the accuracy of widely-cited asteroid data from NASA. Many asteroid experts didn’t take him seriously. But last month, Myhrvold got their attention by publishing his final analysis in a prestigious peer-reviewed journal.

Want to follow the latest news and policy debates over agricultural biotechnology and biomedicine? Subscribe to our free newsletter.

Of course, few scientists have the resources to go to the lengths that Myhrvold did: He hired lawyers to file Freedom of Information Act (FOIA) requests to get NASA to fork over details of its asteroid analysis methods and algorithms.

Nonetheless, showing that there’s a solid scientific basis for questioning the reliability of published findings allows the research community to then evaluate the situation, Steneck said — and it’s an approach that doesn’t require criticizing others or claiming misconduct. Of course, when a field is too tightly controlled by thought leaders, Steneck added, it may be difficult to get dissenting ideas accepted by traditional top-tier journals, but you can probably still publish them in a peer-reviewed open-access journal. If Scientist A is in a position to pursue such a route, then over time, others in the field might come around to the researcher’s view of things.

Alternatively, in certain scenarios, it could make sense to try a less conventional approach: bringing the case to a “scientific sleuth” with a track record for shining light on faulty findings. As Oransky noted, some data detectives have had “a lot of success because they’re careful, they’re measured, they don’t accuse people of misconduct.” These research watchdogs show their analyses of the problem and just say, “This is a pretty serious error, I think the world should look at it,” he explained. They start off by writing polite letters to the authors of the questionable work, and their institutions, but if those efforts go nowhere, the sleuths then take their concerns public by blogging or going to journalists. The overall strategy can be “very powerful” in achieving retractions and corrections, Oransky said.

Otherwise, another way to call out suspect results is to use online post-publication peer review forums, such as PubPeer.com. But scientists who post comments there have to be meticulously careful, Oransky noted, “because it will backfire the minute you make a mistake and someone pounces on it.”

Hopefully, these perspectives provide some useful ideas for Scientist A in considering how to challenge the two colleagues. Frustration and pushback are likely no matter the approach, but a parting comment from Oransky seems worth bearing in mind: When you criticize other scientists’ work, he told me, they’ll probably react in the same ways that you would, if the roles were reversed.

Bottom line: Treat your colleagues as you would wish to be treated yourself.

Ingfei Chen is a California-based writer whose stories have appeared in publications including Scientific American, The New York Times, and Spectrum. She is a former Knight Science Journalism fellow at MIT.

A version of this article was originally published on Undark’s website asScientific Whistleblowing: When Should a Researcher Call for a Retraction?and has been republished here with permission.

The GLP featured this article to reflect the diversity of news, opinion and analysis. The viewpoint is the author’s own. The GLP’s goal is to stimulate constructive discourse on challenging science issues.

Share via
News on human & agricultural genetics and biotechnology delivered to your inbox.
Optional. Mail on special occasions.
Send this to a friend