published 6/28/10

Ioannidis: Making Medical Science Look Bad Since 2005

A famous scientific paper with an irresponsible and misleading title

by Paul Ingraham, Vancouver, Canada MORE
close
Credentials and qualifications

I am a writer and retired Registered Massage Therapist (unusually well-trained for a massage therapist, a 3000-hour program). I’m almost done with a Bachelor of Health Sciences degree. I am a peer reviewer for The Natural Standard, and a copyeditor for Science-Based Medicine. My most important qualification is more than a decade of workaholic post-graduate study, clinical experience, and constant conversations with readers from around the world, including many experts who have provided countless suggestions and criticisms.

For more information, see: Who Am I to Say? More information about my qualifications, credentials and professional experiences for my readers and customers.

In 2005, PLoS Medicine published a now-famous paper with the attention-grabbing but exasperating title, “Why Most Published Research Findings Are False”.1 It was written by John Ioannidis, MD, PhD, an American doctor from Greece, a distinguished author of about 400 peer-reviewed papers, 40 books and book chapters or so, and much more. Ioannidis is a giant, and I am really nobody to criticize.

But I hate the title of that paper!

This intensely intellectual article — it’s completely, hopelessly nerdy — became one of the most downloaded articles in the history of the Public Library of Science and was described by the Boston Globe as an instant cult classic.

I like the content of the paper very much, but the title is a misleading generalization that plays right into the hands of quacks and charlatans, who like nothing better than to cite any expert who seems to be saying that science is so flawed — that “science is wrong” — that it can’t be used to debunk their nonsense. This is their sole interest in the paper, and indeed it has been extensively ~~cited~~ co-opted for this very purpose. A substantial portion of its popularity is probably due to how effectively its title can be used to undermine the credibility of science.

Ironically, while this work should serve to improve the quality of scientific medicine, it is being used by some cranks to attack the scientific basis of medicine.

Dr. Steven Novella, from an article quite similar to this one, Are Most Medical Studies Wrong?

I don’t believe that it’s an option for scientists to be oblivious to these issues — it’s naïve to think that a title like that won’t be abused, and it’s naïve to think it doesn’t matter.

It will be abused, and it does matter.

Science is not wrong

Attention: the paper does not say that “science is wrong”!

What Ioannidis really says is much less ominous: he argues that it should take rather a lot of good quality and convergent scientific evidence before we can be reasonably sure of a “scientific fact,” and he presents good (scientific!) evidence that a lot of so-called conclusions are premature, not as “ready for prime time” as we would hope.2 It’s an extremely important point to make, because science thrives on constantly refining its own methods.

But, and this is my point — it’s okay for science to work like this. Ioannidis did not mean that science is broken or deeply flawed.

In fact, this is actually how science should work, and it’s not the least bit surprising to good scientists, who never claimed in the first place that their results are infallible or that their conclusions are “true.” When most scientists publish, they’re just saying, “Hey, here’s what we did, here’s what we saw — what about you guys?” And then the research community responds, hopefully, with thoughtful criticism, and more research.

“Scientific investigation is the noblest pursuit,” Ioannidis writes.3 “I think we can improve the respect of the public for researchers by showing how difficult success is.”

Sadly, not all scientists are good at their jobs.

A medical Madoff

Credit for that excellent heading goes to Scientific American, which recently reported on “a medical Madoff,” a doctor who published several influential but entirely fraudulent scientific papers:

Over the past 12 years, anesthesiologist Scott Reuben revolutionized the way physicians provide pain relief to patients undergoing orthopedic surgery for everything from torn ligaments to worn-out hips. Now, the profession is in shambles after an investigation revealed that at least 21 of Reuben's papers were pure fiction, and that the pain drugs he touted in them may have slowed postoperative healing.

A Medical Madoff: Anesthesiologist Faked Data in 21 Studies, Borrel (Scientific American)

The story of Scott Reuben’s fraud will also be co-opted to make the point that “mainstream” medical science is rotten. But it wasn’t “science” that was bad, it was Scott Reuben. The “science” wasn’t misleading, Scott Reuben was misleading. Scott Reuben was a liar-liar-pants-on-fire — and thank goodness science is self-correcting by nature, or who knows how much longer he would have gotten away with it.

Kurt Donsbach is going down … finally

It was extraordinary that Scott Reuben managed to get away with publishing fabricated science for a decade — scientific fraud is usually shut down more quickly. On the other hand, alternative medicine doesn’t condemn its own, and con-artists can thrive for decades as long as they distance themselves from real medicine and claim that science does not apply to them.

Kurt Donsbach, a naturopath and a chiropractor, practiced with impunity for half a century before he finally faced eleven felony counts, including “unlawfully dispensing drugs as a cure for cancer” and “falsely representing a cure for cancer” — but he’s one of the few of his kind who even get charged. Most get away with it, largely because they claim to be above scientific criticism.

Reuben’s bogus results were more or less doomed to be debunked. Science usually exposes scientific frauds quite quickly, and this is a much more important characteristic of science than the fact that frauds occur in the first place. Like a bad organ transplant, the scientific community will reject frauds the moment they are detected. This is strikingly different than the world of alternative medicine, where quacks can get away with selling non-medicine for decades, because you cannot discredit someone who had no credit to begin with — someone who never belonged to any respectable community or adhered to any standards in the first place, someone who may indeed define themselves in opposition to professional communities and standards. You can’t boot someone out of a club they never joined, and so they are “regulated” only by popular opinion and market forces — and our culture basically gives them a pass on proving anything, and if doctors and scientists criticize them it’s just a “conspiracy.”

This is why prosecutions of con-artists in health care are remarkably rare. And it’s why anything that undermines the credibility of science in public opinion is an aid and comfort to quacks.

Human error

Fortunately, outright scientific fraud is relatively rare, and it was not the subject of Ioannidis’ paper. But science is a human enterprise, and people are imperfect (you may have noticed). Even honest, well-intentioned, competent scientists can have strong egos and biases, and even the best can make logical mistakes and become locked into their prejudices (none more famously than Albert Einstein, who produced more good ideas in his 20s than most universities can produce in a decade, but still couldn’t accept the implication of quantum mechanics that “God” does indeed “play dice”). Many scientists play into the hands of critics with overconfident “conclusions” that are all-too easy to point to as examples of scientific inconsistency and imperfection.

This overconfidence is what Ioannidis is correctly warning against — not fraud, just cockiness.

This is why I have criticized so many other scientific papers. Here’s a good example, with the cockines built right into the title: “Radial extracorporeal shock wave therapy (ESWT) is safe and effective in the treatment of chronic recalcitrant plantar fasciitis.”4 The evidence in that paper was nowhere near enough to justify that title. It wasn’t fraudulent, just overconfident.

So it’s ironic that Ioannidis’ title was also brash and irresponsible.

Piling irony on irony, it’s possible Ioannidis may not have even written his own title — an PLoS Medicine editor may have done it. Science writer and broadcaster Flora Graham points out she doesn’t write her own headlines:

The subeditor does that! Writing a good headline is much harder than it looks. I have felt that my carefully measured story has been given an inflammatory or inaccurate headline — and given readers the wrong idea. What if Ioannidis’ editor wrote his headline? His paper cautioning against over-interpretation of evidence ends up with a title that over-interprets his evidence!

Ooooh, that rotten “media”!

Usually it is everybody’s favourite scapegoat, the media — not scientists themselves — who unwisely portray research findings as “conclusions.”

Sure, sure, almost every summary (abstract) of a scientific paper has a “conclusion.” But good scientists know full well that all knowledge is tentative, provisional. They know that even the best evidence is just a piece of an evolving (and fascinating) puzzle. But reporters want to make it look like it’s all a done deal! This exacerbates the public perception that science is unreliable and inconsistent and that scientists, like economists, can never agree on anything, the never-ending battle of the experts. And this is good, rich soil for evangelicals and quacks to sow doubt and dissent about evolution and scientific medicine. The average person is much more prone to getting suckered into buying quackery when scientific medicine is constantly portrayed as chasing its own tail.

So it really troubles me when scientists themselves, even incredibly smart guys like Ioannidis — or his editor? — hand the enemy an intellectual bazooka on a silver platter.

Notes

Ioannidis. PLoS Medicine. 2005. Return to text.
But even Ioannidis’ research findings about the wrongness of research findings may be wrong! And precisely this point has been raised by some critics. In 2007, Goodman and Greenlander wrote: “we agree with the paper’s conclusions and recommendations … but calculating the unreliability of the medical research literature, in whole or in part, requires more empirical evidence and different inferential models than were used. The claim that ‘most research findings are false for most research designs and for most fields’ must be considered as yet unproven.” Ioannidis replied substantively, and the debate goes on … as it should. Return to text.
Ioannidis. PLoS Med. 2007. Return to text.
Gerdesmeyer et al. American Journal of Sports Medicine. 2008. This overconfidently titled paper essentially declares that there is no longer any controversy abotu ESWT for plantar fasciitis. However, my confidence in their conclusions is greatly mitigated by the fact that the researchers are on the payroll of a company that makes ESWT devices, and the entire study was funded by that company. For more analysis, see The ESWT Saga Continues. Return to text.