Everything they tell you is wrong

[firecat]

http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/8269/

Ioannidis...and his team have shown, again and again, and in many different ways, that much of what biomedical researchers conclude in published studies—conclusions that doctors keep in mind when they prescribe antibiotics or blood-pressure medication, or when they advise us to consume more fiber or less meat, or when they recommend surgery for heart disease or back pain—is misleading, exaggerated, and often flat-out wrong. He charges that as much as 90 percent of the published medical information that doctors rely on is flawed....Yet for all his influence, he worries that the field of medical research is so pervasively flawed, and so riddled with conflicts of interest, that it might be chronically resistant to change—or even to publicly admitting that there’s a problem.

The article is infuriating because it uses the same kinds of exaggeration that it's criticizing. ("one large randomized controlled trial even proved that secret prayer by unknown parties can save the lives of heart-surgery patients, while another proved that secret prayer can harm them." Um...I believe that individual scientific studies themselves don't pretend to "prove" anything?)

But I do agree with the point that, due to biases and pressures built into the system, the results of medical research are not reliable.

Comments

[flarenut]

Not that it matters so much, but I think (and of course I would) that the kind of hyping the article does is of a different, mostly less pernicious kind than what it decribes. Between publication bias, bad design, post-hoc cherrypicking and all the other tricks, you can get completely bogus results described in the most sober, apparently objective, thoroughly convincing fashion you could ask for.

I think Ioannidis also misses an important point about how information works. If you confirm some item of common belief, you're not adding (in an information-theoretic sense) much to the world's knowledge. If you disconfirm or show something new, you're adding a lot more, and it's only reasonable that people will pay more attention. Of course, something new or disconfirming is more likely to be wrong, but ideally there would be checks and balances.

And of course the key rule of thumb in all of this: one out of every 20 random sequences will produce a result that is significant at the 95% level.

[firecat]

Those are really good thoughts.