SaveYourself.ca helps you solve pain problems

published 03/11/09

Bogus Citations

References to “scientific evidence” are routinely misleading

by Paul Ingraham, Vancouver, Canada MORE
close

Credentials and qualifications

I am a writer and retired Registered Massage Therapist (unusually well-trained for a massage therapist, a 3000-hour program). I’m almost done with a Bachelor of Health Sciences degree. I am a peer reviewer for The Natural Standard, and a copyeditor for Science-Based Medicine. My most important qualification is more than a decade of workaholic post-graduate study, clinical experience, and constant conversations with readers from around the world, including many experts who have provided countless suggestions and criticisms.

For more information, see: Who Am I to Say? More information about my qualifications, credentials and professional experiences for my readers and customers.


Recently I got an email from a reader who wanted to make a point he thought I might be interested in (always welcome). He included a referred to a scientific paper to support for his point — even more welcome, and quite unusual. Most people are not this diligent.

Unfortunately for his credibility, I actually checked the reference — imagine! — and I quickly realized that the paper did not actually support his point, and this wasn’t ambiguous. It wasn’t really a matter of interpretation or opinion. It was way off: he had read much more into the paper than the researchers had ever intended anyone to get from it. The authors had clearly defined the limits of what could be interpreted from their evidence.

Even professionals do this.

The phenomenon of the bogus citation is almost a tradition in both real science and (especially) in science reporting. I have often across full-blown irrelevancy in citations, and dubious relevancy is epidemic. Real live scientists — not the good ones, of course, but scientists nevertheless — often “pad” their books and articles with barely relevant citations. They probably pulled the same crap with their term papers as undergrads!

Real live scientists often “pad” their books and articles with barely relevant citations.

This kind of stuff gives science a bad name. But it’s not that science is bad — it’s just bad science. This is why nothing can ever be “known” until several lines of research have all converged on the same point.

So, word to the wise: citations are not inherently trustworthy.

How to Refer to Scientific Evidence
Right! Wrong!
refer to relevant, good-quality evidence published in a respectable peer-reviewed scientific journal the sneaky reach: make just a bit too much of good and relevant evidence … without even realizing it yourself, probably
the big reach: make waaaaay too much of otherwise good evidence
the curve ball: reference to perfectly good science that has little or nothing to do with the point
the bluff (A.K.A. “the name drop”): citation selected at random from a famous scientific journal like The New England Journal of Medicine (because no one actually checks references, do they?)
the ego trip: cite your own work … which in turn cites only your own work … and so on …
the curve ball: referencing research that perfectly supports your point, but is published by hacks and quacks in a crap journal no one’s ever heard of or ever will again
the uncheckable: citing a chapter in a book no one can or would ever want to actually read, because it has a title like Gaussian-Verdian Analysis of Phototropobloggospheric Keynsian Infidelivitalismness … and it’s been out of print for decades and it’s in Russian
the really low road: just make stuff up!

To trust a scientific reference without checking yourself, you have to really trust the author. But you can only trust an author by actually checking their references a few times. And, even after trust is established, you should probably still check the occasional reference. And always check references when the truth actually matters to you.

This is why I have gone to considerable technological lengths on SaveYourself.ca not just to cite my sources for every key point, but also to provide user-friendly links to the original material … which makes them easy to check! This is basic principle of responsible publishing of health care information online — if you’re not going to leverage to technology to facilitate reference-checking, why even bother?

A perfect example: SleepTracks.com tries to sell a dubious product with bogus references to science

I was making some corrections to my insomnia tutorial when I curiously clicked on an advertisement for SleepTracks.com, where a personable blogger named “Yan” is hawking an insomnia cure: “brain entrainment” by listening to “isochronic tones,” allegedly superior to the more common “binaural beats” method.

Yan goes to considerable lengths to portray his product as scientifically valid, advanced and modern, and he actually had me going for a while. He tells readers that he’s done “a lot of research.” To my amazement, he even cited some scientific papers. I had been so lulled by his pleasant writing tone that I almost didn’t check ‘em.

“Here are a few sleep-related scientific papers you can reference,” Yan writes, and then he supplies these three references:

Notice anything odd there? They lack dates. Hmm. I wonder why? Could it be because they’re from the stone age?

Those papers are from 1967, 1981, and 1982. In case you’ve lost track of time, 1982 was almost thirty years ago — not exactly “recent” reseach, particularly when you consider how far neuroscience has come in the last twenty years. Now, old research isn’t necessarily useless, but Yan was bragging about how his insomnia treatment method is based on modern science. And the only three references he can come up with pre-date the internet by a decade? One of them pre-dates me.

So, how much modern scientific research is there about isochronic tones and neural entrainment as a treatment for insomnia?

Search PubMed (the biomedical database) for “isochronic insomnia” — click this for the search — a nice broad search that should catch any paper that even refers to Yan’s pet theory. How many results are there?

Zero!

Clearly, these are references intended to make him look good. Yan didn’t actually think anyone would look them up. Yan was wrong. I looked them up. And their age isn’t the worst of it.

The 1981 study had negative results. The biofeedback methods studied — which aren’t even the same thing Yan is selling, just conceptually related — didn’t actually work: “No feedback group showed improved sleep significantly.” Gosh, Yan, thanks for doing that research! I sure am glad to know that your product is based on thirty year-old research that showed that a loosely related treatment method completely flopped!

The 1982 study? This one actually had positive results, but again studying something only sorta related. And the sample size? Sixteen patients — a microscopically small study, good for proving nothing.

The 1967 study? Not even a test of a therapy: just basic research. Fine if you’re interested in what researchers thought about brain waves and insomnia before ABBA. Fine if you want to include numerous other references. But as one of three references intended to support the efficacy of your product? Is this a joke?

So Yan gets the Bogus Citation Prize of the Year: not only are his citations barely relevant and ancient, but are obviously deliberately cited without dates to keep them from looking as silly as they are.


Further Reading