Everything they tell you is wrong

[firecat]

http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/8269/

Ioannidis...and his team have shown, again and again, and in many different ways, that much of what biomedical researchers conclude in published studies—conclusions that doctors keep in mind when they prescribe antibiotics or blood-pressure medication, or when they advise us to consume more fiber or less meat, or when they recommend surgery for heart disease or back pain—is misleading, exaggerated, and often flat-out wrong. He charges that as much as 90 percent of the published medical information that doctors rely on is flawed....Yet for all his influence, he worries that the field of medical research is so pervasively flawed, and so riddled with conflicts of interest, that it might be chronically resistant to change—or even to publicly admitting that there’s a problem.

The article is infuriating because it uses the same kinds of exaggeration that it's criticizing. ("one large randomized controlled trial even proved that secret prayer by unknown parties can save the lives of heart-surgery patients, while another proved that secret prayer can harm them." Um...I believe that individual scientific studies themselves don't pretend to "prove" anything?)

But I do agree with the point that, due to biases and pressures built into the system, the results of medical research are not reliable.

Comments

[submarine_bells]

Reading through the article, I think that a lot of the exaggeration that you refer to is simply the problem of having scientific stuff written up by a non-scientist in a journalistic style, where the normal caveats that one might hope to find (or might be uttered by the source being interviewed) will mostly get overlooked or rewritten in a more "exciting" style for a general audience. With that in mind, and reading the article with that sort of filter in place, it's not bad at all. Ioannidis is quite correct. I do sometimes wonder if training in research methodology is a standard part of the training that a lot of folk who go on to do medical research receive, since I've seen quite a lot of methodologically disastrous medical studies in publication. It's a worry, and I'm glad that the issue's getting a bit of "air time".

[firecat]

I think it's possible to write in an exciting style without being inaccurate or overexaggerating, but I might be wrong...I'm certainly not getting published in The Atlantic. :-)

[flarenut]

Not that it matters so much, but I think (and of course I would) that the kind of hyping the article does is of a different, mostly less pernicious kind than what it decribes. Between publication bias, bad design, post-hoc cherrypicking and all the other tricks, you can get completely bogus results described in the most sober, apparently objective, thoroughly convincing fashion you could ask for.

I think Ioannidis also misses an important point about how information works. If you confirm some item of common belief, you're not adding (in an information-theoretic sense) much to the world's knowledge. If you disconfirm or show something new, you're adding a lot more, and it's only reasonable that people will pay more attention. Of course, something new or disconfirming is more likely to be wrong, but ideally there would be checks and balances.

And of course the key rule of thumb in all of this: one out of every 20 random sequences will produce a result that is significant at the 95% level.

[firecat]

Those are really good thoughts.

[cleverthylacine]

It absolutely is. But there are trends in what does and does not get published. A few years ago I was seeing a lot of articles in magazines about how it was more important to eat well, get enough sleep and exercise some than it was to be under a certain weight. You never see that these days; you see all this scare stuff about how "obese" people are all going to die, even though epidemiologically, people at or below what's considered "normal" weight die sooner than those who are somewhat above it. (Being very large is a risk factor, but I've never seen a single one of those studies risk-adjusted to take into account physician bias, and how doctors assume that everything that is wrong with a fat person is because of their weight, which causes necessary testing to be delayed and drugs to be withheld until problems have become much more serious.)

A few years ago, there was all this freakage about how we should assume people are bipolar first, because antidepressants can make people who are somewhat but not very bipolar go manic. Now they're starting to realise that you really don't want to put anyone on an antipsychotic unless you're absolutely sure they need one, because a manic episode can be fixed, but some of the antipsychotic side-effects are horrible and hard to get rid of. :/

And there are a lot of magazines that pounce all over this stuff. You can count on any woman's fashion magazine (with the exception of those marketed for plus women) to be all over the fat scare/fat hate stuff. "Childhood obesity" sells too.

[firecat]

But there are trends in what does and does not get published.

There sure are.

I think my "favorite" was "this generation of children will have shorter lives than their parents, because of obeeeeesity." Especially when it showed up in the Parks & Rec newsletter of the town I live in.

[cleverthylacine]

so much lolwut.

But of course this is on a P&R newsletter because they want to scare parents into sending their kids there. :(

[firecat]

I am tempted to go around town wearing the statement I AM THE PEOPLE YOUR PARENTS WARNED YOU ABOUT.

[cleverthylacine]

As someone who works in medical research I think that the research itself is usually reliable, but that people only hear about the research that the people currently in power want them to hear about, and research sometimes doesn't get done because no-one will fund it--it's difficult to get the government to fund research on obesity that has nothing to do with weight loss and can't be used to scare people into dieting, and you can't often get permission to study marijuana and other schedule I drugs if you aren't trying to prove that they're bad.

[firecat]

Yep that's certainly a huge part of the problem with the system.

[cleverthylacine]

But I really do blame the media for a lot of the bad information "from medical research" that gets out there, because reporters know that badly rewriting a scary study (that often is not corroborated by other studies) is a quick way to get attention. :(

[firecat]

That's another huge part of the problem...

[lorres]

I think there's also probably a breakdown in medical practice as it relates to the research. If a doctor has read x study or x studies indicating that a certain treatment ends in positive results for certain symptoms, the doctor may be good or less good at judging whether the case in hand is a case that would probably be treatable in the same way the study(ies) indicate.

In my mind there's more work to be done in medical practice - how to use research knowledge that is come by in carefully controlled experiments in the treatment of real humans that are not statistically abstracted entities but individuals of complex mind/body being. And of course I believe that nasty side effects are way too often ignored by both researchers and practitioners.

[flarenut]

My favorite on the side effect thing was a study someone did (a drug company, but even so) on the relative experiences of doctors and patients regarding migraine treatment. Patients who thought their doctor wasn't listening to their descriptions of side effects or effectiveness would stop going to that doctor. Doctors whose patients stopped coming to them would figure that the patient's problem had gone away.