July 14, 2005


Research: Third of study results don't hold up (AP, 7/13/05)

New research highlights a frustrating fact about science: What was good for you yesterday frequently will turn out to be not so great tomorrow.

The sobering conclusion came in a review of major studies published in three influential medical journals between 1990 and 2003, including 45 highly publicized studies that initially claimed a drug or other treatment worked.

Subsequent research contradicted results of seven studies -- 16 percent -- and reported weaker results for seven others, an additional 16 percent.

That means nearly one-third of the original results did not hold up, according to the report in Wednesday's Journal of the American Medical Association.

What if this study is the one in three?

Posted by Orrin Judd at July 14, 2005 5:00 PM

33-1/3% chance of that. An obviously unacceptable risk.

Posted by: ghostcat at July 14, 2005 5:25 PM

True. It could be that 100% of studies don't hold up.

Posted by: pj at July 14, 2005 5:34 PM

I've switched between butter and margarine so many times, I forgot which one I like better.

Posted by: erp at July 14, 2005 6:18 PM

This study should be mathematiclly correct, but you have to assume the studies impeaching the prior studies were correct. Hmm, this gets complicated.

Posted by: jdkelly at July 14, 2005 6:30 PM

this calls for a meta-study

Posted by: cjm at July 14, 2005 6:35 PM

In recent months alone, the scientific community has rescinded a number of previously proclaimed findings.

It now appears that people who are somewhat over the government's ideal weight classification actually die at rates lower than folks who are under or precisely on that previous ideal weight. For years, medical experts preached that the dehydration was the biggest threat to long distance runners, but they now say injesting too much water actually contributes to more deaths than does dehydration. And of course, the same folks who now push the global warming message once warned that nuclear winter would be the result of the exact same forces.

Science is great. But our scientists, the policy makers who use that science, and the media who trumpet these often-wrong "findings" could all benefit from a good strong dose of humilty.

Posted by: jsk at July 14, 2005 8:10 PM

This is roughly consistent with an observation I made in graduate school back in the 1980s, that people tend to be about a factor of 10 optimistic about the statistical significance of their results. The standard in science is that there should be no more than a 5% probability that your results are due to chance. Ten times that is 50%, not far off 33.3%. Of course, there's nothing wrong with doing better than 5%, which, even with the factor of 10, would give you better than even odds.

Note that fields where you can do much better than 5% make much faster progress than fields where you often can't do much better. If those 5% studies are really 50/50, it would explain a lot.

(I wrote a paper in grad school that was never published. My professor wanted to include some results that looked real impressive, but after applying a factor of 10 to the statistics, I refused to go along. He really wasn't happy with me on that one.)

Posted by: Bob Hawkins at July 14, 2005 10:40 PM

Doctors and computers. More dangerous than automatic weapons.

Posted by: Robert Schwartz at July 14, 2005 11:45 PM

Where human health and behavior are concerned, there are too many variables that can have an impact. These studies are trying to give an up or down judgement on a single substance or behavior or environmental factor without taking into account how that factor interacts with all the other factors. There is no way to control for all of the variables in a study.

Not to mention the underlying bias of the ones doing the study. They are either funded by government or interest group nannies to show that such and such is not good for you, or are funded by an industry group to show that such and such is good for you. It is important to look at the actual results of the study and not the spin that was put on it by the sponsors or the media. The media is especially prone to exaggerating the results of a study in order to catch your attention in a soundbite or headline.

Posted by: Robert Duquette at July 15, 2005 8:54 AM

90% of statistics are made up.

Posted by: at July 15, 2005 5:59 PM

This includes the secondhand smoke studies where even more turned out not to be conclusive.

Posted by: Syl at July 15, 2005 6:53 PM

90% of statistics are made up, including this one.

Posted by: Robert Schwartz at July 17, 2005 7:20 PM