published 8/28/04

Extraordinary Claims

A guide to critical thinking, skepticism and smart reading about health care on the web

by Paul Ingraham, Vancouver, Canada MORE
close
Credentials and qualifications

I am a writer and retired Registered Massage Therapist (unusually well-trained for a massage therapist, a 3000-hour program). I’m almost done with a Bachelor of Health Sciences degree. I am a peer reviewer for The Natural Standard, and a copyeditor for Science-Based Medicine. My most important qualification is more than a decade of workaholic post-graduate study, clinical experience, and constant conversations with readers from around the world, including many experts who have provided countless suggestions and criticisms.

For more information, see: Who Am I to Say? More information about my qualifications, credentials and professional experiences for my readers and customers.

Bad information is the chief hazard of the Information Age. The glut of unregulated information about your health available on the world wide web constitutes both an incredible opportunity and a real threat to your safety and your pocketbook. In the absence of any publication standards, how can you tell the difference between the good and the bad? There are three major signs of poor quality in health information:

extraordinary claims
author hopes to sell you something
no evidence or references offered

Let’s look at those in some more detail …

Bad Sign No. 1: Extraordinary claims

An interesting debate has gone on … between those who think that all doctrines that smell of pseudoscience should be combated and those who believe that each issue should be judged on its own merits, but that the burden of proof should fall squarely on those who make the proposals. I find myself very much in the latter camp. I believe that the extraordinary should be pursued. But extraordinary claims require extraordinary evidence. [Italics mine.]

Broca's Brain, by Carl Sagan (p62, 1972 hardcover edition)

Carl Sagan was the Twentieth Century’s champion of public education about science, the scientific method, and critical thinking. His statement about extraordinary claims is perhaps the most famous of all comments ever made about skepticism, and it is the most important piece of wisdom an information warrior needs in the Twenty-First Century.

An extraordinary claim is any assertion that seems inconsistent with how many informed people understand the world. There is nothing necessarily wrong with an extraordinary claim, but it is necessary to set a higher standard in supporting it. In particular, anyone selling anything or promoting any idea that sounds too good to be true needs to offer not just evidence, but unusually good evidence — testimonials are not enough!

There is nothing necessarily wrong with an extraordinary claim, but it is necessary to set a higher standard in supporting it.

The history of science is full of examples of extraordinary claims that turned out to be correct.

“The world is round” was once an extraordinary claim!
“Water is full of bugs too small to see” was dismissed as the ravings of a madman — until enough biologists had a chance to a look through a microscope for themselves.
“Surgeons must wash their hands” nearly resulted in Joseph Lister’s professional excommunication — but he was right, and the evidence was exceptional and unavoidable.

History is also full of extraordinary claims that went nowhere. To this day, several popular pseudosciences and superstitions persist on the fringes of society, and their advocates complain that they are not taken seriously — but they remain marginalized because they do not present extraordinary evidence, or any at all, to support their extraordinary claims.1

History is full of extraordinary claims that went nowhere.

The most obvious kind of extraordinary claim in health care is the panacea, a treatment or medicine that is supposed to be good for anything that ails you. Here are some examples:

Upper cervical chiropractors claim to be able to treat a wide range of pathologies by manipulation of only the top two joints of the spine.
F. Batmanghelidj, MD, claims that drinking a lot of water (and buying his book, and taking his workshop) can cure “heartburn, arthritis, back pain, angina, migraines, autoimmune diseases, colitis, asthma, high blood pressure, diabetes.” Wow! This is a particularly classic panacea.2
So-called anti-depressant medications like Prozac, Paxil and Zoloft prescribed like a panacea for an inappropriately wide variety of conditions.3
Reflexologists claim to be able to treat any organ or system in the body by physical manipulation of only the hands, feet and ears.

Panaceas are usually obvious. Unfortunately, health care is also full of extraordinary claims that are often mistaken for ordinary claims. A combination of widespread public biological illiteracy makes it easy for ignorant and/or deceitful people to make bad promises that seem perfectly reasonable to the average person. There is no easy defense against this except a high degree of caution.

Bad Sign No. 2: Selling (ka-ching, ka-ching!)

Health care information and the profit (or prestige) motive are not usually good bedfellows.

However, it is not wrong in principle to profit from genuinely valuable health care services, products, or information. In fact, it’s necessary for those who have invested in training, product development or publication to promote themselves and earn a financial return — if what they sell is actually beneficial.

It is not wrong in principle to profit from genuinely valuable health care services, products, or information.

Unfortunately, it’s always been easy to make a profit from selling health care that is not beneficial, or only partially so. Public ignorance about science in general and biology in particular is epidemic. Worse still, people who are sick or hurt are eager to believe in anything that might help ease their pain. Since the only possible way to sell snake oil is to present deceitful and/or irrelevant information, there is an awful lot of that available.

The only solution for this is education — the buyer must be aware and critical. At the least, bogus health care products, services, and advice can waste your time and money. At worst, they may harm you. Never purchase unless you have been convinced by good quality evidence. Don’t believe anything you are told by someone with a profit (or prestige) motive unless they support their claims responsibly.

Beware in particular of the most insidious threat of all: the salesman who persuades you to buy an ineffective product or service by giving you good information … about something else! Sneaky, eh? It happens all the time. Indeed, this is a favourite tactic of pharmaceutical manufacturers, who promote their reputations and increase sales of their products by providing perfectly good information to consumers and doctors that has nothing to do with the effectiveness or safety of their products. For example, they provide doctors’ with high quality patient education tools, such as charts — always with a logo on it.

The only possible way to sell snake oil is to present deceitful and/or irrelevant information.

The point is that you must never judge any health care product, service or advice by anything except the evidence for that product, service or advice — and the more someone stands to gain from your belief, the more careful you must be.

It is not just extraordinary claims that require extraordinary evidence — profitable claims require extraordinary evidence!

Bad Sign No. 3: No evidence, no references = information malpractice

Unreferenced medical information is usually next to useless. A health care publication without footnotes is like a doctor without a license to practice. When it comes to information about health, the stakes are simply too high for publishers to ask readers to take their word for anything.

Like references on a resumé it often doesn’t matter if you actually check citations — what matters is that it can be checked, and the more easily the better. Ideally, citations on the web should contains links to abstracts (summaries), or even the full text of the article. Recent publications should generally be preferred over older ones.

A health care publication without footnotes is like a doctor without a license.

A nation’s currency rests on metal — there has to be real gold somewhere, or the value of our money is a mass delusion. In a similar sense, citations to scientific literature is the bedrock of medical information. Every medical idea, every recommended treatment, every new idea has to be based on evidence published in a peer-reviewed scientific journal — and if it can’t be, it had better have a good reason. The greater the risks, the more extraordinary the claim, the more expensive the service … the greater the need for this trail of bread crumbs to the original research.

Not all health care information has to rise to this standard. Some of the articles I publish don’t require a lot of scholarship, for instance, because they ask nothing of the reader: they don’t have to buy anything, pick sides in a controversy, or make a medical decision based on the information. Only when the stakes go up is there a need for evidence.

Unsupported medical opinions are acceptable sometimes — the practice of health care would grind to a halt without them. But opinions are not evidence, must be clearly identified, and opinions that matter more must be more clearly identified. It’s irresponsible for an expert to disguise an opinion as fact, or to carelessly allow opinion to stand as fact in the minds of readers who may take expert opinions as seriously as good evidence. Experts must use cautious language when expressing opinions — such as “in my opinion” or “I believe.”

A document without references or acknowledgements of uncertainty is information malpractice.

A final consideration: the source

You will be reasonably safe from bad information if you expect any significant claim to be supported by evidence, and if you raise the bar when the claim is extraordinary. If you are even more cautious where someone is trying to sell you something, and if you expect to see references at the end of any serious medical document, then you are officially a “critical reader.”

But there is one more important point, and it’s a good one to finish with.

There is no such thing as an unbiased or objective source of information, although some sources are more so than others. I believe that this is a myth perpetuated by journalists and profitable news media who want you to believe that they are a superior source of information. The truth is, everyone has a bias, and media consumers simply need to know what the bias is.

For instance, no one accuses Michael Moore or Rush Limbaugh of being unbiased — but they are good sources of information because we know what their biases are. The worst source of information is an unidentified source, where you do not know and cannot even guess at the biases.

Oddly enough, this is often an advantage of searching for information on the web: the biases of authors are often clear, and it’s often easy to find a wide variety of viewpoints quickly.

The worst source of information is an unidentified source.

Please never be impressed by credentials alone. Credentials can be faked, and they often are. Although credentials may indicate expertise, and give you some idea of what the biases of the author may be, they do not guarantee good information. To be blunt, I believe that credentialled experts are often the worst of the “information offenders.” Unfortunately, it is a strong feature of human nature to rest on one’s laurels, and many professionals don’t seem to think that they need to support their claims.

Happy critical reading to you!

Appendix: What makes a scientific paper so scientific?

A scientific paper really is a different animal than a normal “article.” Most people have never read a scientific paper. Even though I have done a lot of homework in my field, I have probably only read a few dozen scientific papers at most.

What makes a scientific paper special is that there are strict standards for their publication, and clear general guidelines for all scientists about what constitutes good evidence. Publication in a scientific journal is not a guarantee that information is good — like democracy, it’s an imprefect system, but it’s the best we’ve good.

A good scientific paper has the following qualities:

Peer-reviewed. Most scientific journals publish only “peer-reviewed” articles: that is, articles that must be approved by a group of scientists with relevant expertise. This means that you can rest assured that no wacky theories will be published in a scientific journal without first impressing several other scientists with the evidence. (This can also be a limitation, of course — good new ideas are sometimes suppressed.)
Primary research. Scientific journals publish a variety of commentaries and editorials, tutorials and “literature reviews.” These can be useful, but by far the most important kind of scientific article is reporting primary research: direct experiment and study.
Larger sample sizes. Most medical research involves studying a group of people. Generally speaking, the larger the group, the more meaningful or relevant the results of the research. A small sample is 25, and may be adequate depending on the circumstance. Less than that is rarely useful.
Randomization. It’s important that the groups of people that medical scientists study be randomized. The problem is that it’s easy to inadvertently skew results by choosing people who are likely to produce certain results. A classic example, and a mistake no good researcher would make, would be to ask how common back pain is by studying a group of massage therapy clients — the results would have no general significance, because massage therapy clients are much more likely to have back pain than most groups of people. So randomization (by any of several methods) is quite important.
Control. It’s also important that a study group be “controlled” — that is, that is be compared to a neutral group. It is meaningless to study the effect of a surgery for knee injury, for instance, if you don’t compare results to a group of people receiving a fake surgery (a placebo). If you didn’t compare, you could easily make the mistake of thinking that the surgery was effective, when in fact both groups might get better just as quickly. Believe it or not, exactly that research has been done[Moseley] — an it is one of the best demonstrations of the importance of control groups in recent history. Unfortunately, a surprising amount of uncontrolled scientific research gets published — so watch out for this one. Research without a control group doesn’t tell us nearly enough!

So, now you are a seriously informed reader: the next time you see a dubious claim, look for supporting evidence published in a “peer-reviewed scientific journal.” Look it up. Is the paper “primary research”? Is so, was there a decent “sample”, and was it “randomized” and “controlled”?

Notes

Consider the example of Harry Houdini, who sincerely wanted a medium to help him talk to his dead mother — but the master of illusion could not find a medium who was not an obvious fraud to him. Or, consider James Randi, a prominent skeptic and debunker, who has offered a million-dollar prize for the last several years to anyone who can convincingly demonstrate a paranormal ability such as ESP, telekinesis or clairvoyant interpretation of Tarot cards. No challenger has yet won the prize. Return to text.
See Water Fever and the Fear of Chronic Dehydration for more information about this egregious bit of poor quality health care information. Return to text.
Hypericum Depression Trial Study Group. Effect of Hypericum perforatum (St John's wort) in major depressive disorder: a randomized controlled trial. JAMA. 2002 Apr 10;287(14):1807-14, abstract. This good quality study showed that “neither sertraline [Zoloft] nor H perforatum [St. John's Wort] was significantly different from placebo.” See SSRI Antidepressants Are Not Medicine for more detailed opinions on this subject. Return to text.