SaveYourself.ca •Sensible advice for aches, pains & injuries
 

The “Impress Me” Test

Most controversial therapies are fighting over scraps of scientific evidence

450 words, published 2009, updated May 15th, 2013
by Paul Ingraham, Vancouver, Canada bio
I am a science writer, the Assistant Editor of ScienceBasedMedicine.org, and a former Registered Massage Therapist with a decade of experience treating tough pain cases. I’ve written hundreds of articles and several books, and I’m known for sassy, skeptical, referenced analysis and a huge bibliography. I am a runner and ultimate player, and live in beautiful downtown Vancouver, Canada. • full bioabout SaveYourself.ca

Readers and patients are forever asking me what my “hunch” is about a therapy: does it work? Is there anything to it? I’m honoured that my opinion is so sought after, but I usually won’t take the bait. Like Carl Sagan, “I try not to think with my gut.”

It’s okay not to know. It’s okay for the jury to be out.

And it had better be, because there’s still a great deal of mystery in musculoskeletal health science. Most of the scientific evidence that I interpret for readers of SaveYourself clearly fails the “impress me” test. Even when that evidence is positive, it’s often only sorta positive. Even when there’s evidence that a therapy works, it’s usually weak evidence: some studies concluded that maybe it helps some people, some of the time … while other studies, almost always the betters ones, showed no effect at all. I’m supposed to get excited about this? To justify confidence in a therapy, we want really good evidence, evidence that makes you sit up and take notice, evidence that ends arguments because it’s just that clear.

Anything less is kind of lame. Anything less is underwhelming. Anything less fails to impress!

The evidence is also often just preliminary. You can critically assess it (and I do!), but “replication needed" is the ultimate caveat in scientific criticism. It covers all the bases. Everything else is just details. At the end of the day, if promising results cannot be replicated by other researchers, it doesn’t really matter what was wrong with the original research. Either a treatment works well enough to consistently produce impressive results … or it doesn’t.

Thus, “controversy” about many popular therapies is much ado about nothing, and proponents are getting almost hysterical fighting over scraps of evidence. After decades of study, the effectiveness of a therapy should be clear and significant in order to justify its continued existence or “more study.” If it’s still hopelessly mired in controversy after so many years — more than a century in some cases — how good can it possibly be? Why would anyone — patient or professional — feel enthusiastic about a therapy that can’t clearly show its superiority in a fair scientific test? Where’s the value in even debating a therapy that is clearly not working any miracles, that has a trivial benefit at best?

Science, as they say, really delivers the goods: missions to Mars, long lives, the internet. A therapy has to deliver the goods. It’s got to help most people a fair amount and most of the time … or who cares?

Until it impresses you, it’s just some idea that hasn’t yet showed much promise.

We must somehow find a way to make peace with limited information, eagerly seeking more, without being dogmatic about premature conclusions.

Science Science And The Game Of 20 Questions, by Val Jones

I don’t want to believe. I want to know.

Carl Sagan