Limits of the scientific method

[firecat]

"The Truth Wears Off: Is there something wrong with the scientific method?" by Jonah Lehrer

...all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology.

Comments

[lilacsigil]

I like this article very much, because it proves we're not smarter than people of previous generations - we just have more theories which have had less time to be thoroughly tested. The people in medical trials for specific condition, for example, are often people who cannot get drugs any other way or who are the sickest, because their doctors will get them enrolled in a desperate bid to help them. Hooray, great results! Then it goes out to the general populations and results decline as moderately and mildly sick people (and misdiagnosed people) use the same drug in large numbers. Even in the last few years, I remember Zyprexa being the magical wonder drug - the truth is that it works extremely well for some people, moderately for some, and poorly for others, just like all wonder drugs!

[firecat]

the truth is that it works extremely well for some people, moderately for some, and poorly for others, just like all wonder drugs!

Very good explanations for why this happens with medical treatments, yeah.

[eagle]

That's an excellent article. Thank you for the link! It manages to take on a very important subject that undermines a lot of reported scientific research without being anti-science, which is difficult to do and very well-handled.

It's also very kind, which is a good thing since hopefully that will make it more persuasive. It goes to great length to not assume malice. But it left me with the desire to say, somewhere for the record, that I firmly believe many of the problems with reproducibility of medical studies in particular is not due primarily to subtle effects of selective reporting, confirmation bias, or inappropriate handling of randomness. It's due to widespread, outright, pre-meditated fraud by the pharmaceutical companies, including falsification of results, intentional selective screening of results, bribing of journals, construction of fake journals, and similar sadly straightforward basic tactics of corruption.

[firecat]

Well, yeah. I completely agree.

I posted a while back about an article focusing on Dr. Ioannidis (mentioned in the New Yorker article). It discusses some of the less subtle pressures that skew data, although doesn't discuss fraud so much.

http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/8269/

[bcholmes]

Good article. Thanks for the link.

Limits of the scientific method

[betonica]

Thanks! Someone else posted this somewhere recently - I thought it was a fellow faculty member in email, but can't find it. Anyway, I agree with both possible factors: that there are subtle things that have made initial studies more convincing than later ones, and that there is outright fraud.

Also, though, the placebo effect appears to be growing: http://www.wired.com/medtech/drugs/magazine/17-09/ff_placebo_effect?currentPage=all - so, for medical trials at least, it isn't that the drugs are somehow becoming less effective (or were never that effective in the first place but were reported erroneously), but what they're measured against has gotten more powerful.

I'm all for the placebo effect. As I tell my students, it isn't that you're just *thinking* that you're getting better but not, it's that you're actually *getting* better, due (in part, one assumes) to thinking that you will. Excellent medicine with few side effects. And inexpensive. What's not to love?

But the overall issue with the "truth wearing off" is certainly an interesting puzzle, and may not be just due to the above three factors. I love it when science gets tweaked, and absolute truths become puzzling uncertainties.

Re: Limits of the scientific method

[firecat]

What's not to love?

Giving patients a placebo without their knowledge is sometimes not to love, IMO. But recent studies suggest that the placebo effect might work even when no deception is involved. That's promising.

It would be pretty cool if we could figure out how the placebo effect works.

[flarenut]

To a first approximation, one out of every 20 studies reporting something confirmed at a 95% level will be reporting on complete randomness. And there are a lot of studies published...

[e4q.livejournal.com]

i am glad that gravity is still true.

and that scientists are having to keep on their toes.

[firecat]

Electromagnetic theory is working pretty well here, at the moment.

[e4q.livejournal.com]

handy.

also, a lot of my drugs are still working, and i am finding a lot of stuff like bricks are still holding together, and cement is still good, steel, radio waves. all good.

[elainegrey.livejournal.com]

There's a reason the social scientists get eye rolls from the physical scientists. Too many uncontrolled variables, too small sample sets, to short a duration.

*grumble*

Not that physical science shows a blessed perfection. I can't remember the constant that was (mis)predicted and the huge error bars folks had on their measurements to include that value, and then eventually the value was recalculated and *ping* the error bars shunk.

[vito-excalibur.livejournal.com]

There is 100% something wrong with Mr. Lehrer's understanding of the scientific method, but that is not the same thing!

The steady decline in observed phenomena from initial experimental results has been a known issue for a long time. The issue is that scientists fudge their results, knowingly and not. They frequently just do not publish results that differ too much from the previous literature in the field. So someone gets an exciting result, publishes it, it is talked about as the Next Big Thing. Someone else tries to replicate the experiment. They do not see that result - but there are always error bars - they end up publishing a smaller result. The next person does the same thing, etc., and you get a slow and embarrassing regression to the mean. But that is science working; i.e., making errors, checking on them, and correcting them.

[firecat]

There may be something wrong with the title of the article. The article discusses most of the things you mention.

[cakmpls.livejournal.com]

Yes, all of that is pretty much what the article says!

Done yesterday (20100110 Mo)

[pingback-bot.livejournal.com]

User mdlbear referenced to your post from Done yesterday (20100110 Mo) (http://mdlbear.livejournal.com/1300490.html) saying: [...] via firecat [...]