BECAUSE SCIENCE IS HUMAN:

Most medical research is wrong? Are you kidding?: No. In fact, the leading figure in medical statistics says plainly, “most claimed research findings are false”. (Michael Cook, 11/22/10, MercatorNet)

Dr Ioannides is not a crank or an enemy of science. On the contrary, his work has been published in leading journals and his claims are widely accepted among his colleagues. He has worked at Harvard University, Tufts University and Johns Hopkins University. His ground-breaking 2005 paper in the journal PLoS Medicine has become the most downloaded in its history. Every year he receives hundreds of invitations to speak at conferences. “You can question some of the details of John’s calculations, but it’s hard to argue that the essential ideas aren’t absolutely correct,” Doug Altman, the director of Oxford’s Centre for Statistics in Medicine, told Atlantic Monthly.

Ioannides’s claims are largely statistical and thus require much brain cudgelling for laymen. But his conclusions ought to rattle anyone: that “most research findings are false for most research designs and for most fields” and “claimed research findings may often be simply accurate measures of the prevailing bias”.

Why is this?

There are a number of interlocking reasons. Many studies are too small to be reliable. The best ones involve several thousand subjects, but many studies, especially in genetics, are based on fewer than a hundred. Many studies are badly designed or are hard to compare to other studies of similar data.

Prejudice plays a role as well. It’s not necessarily ideological or financial; old-fashioned chest-beating, turf-protecting arrogance is just as effective. Scientists who are committed to a theory are less likely to find contradictory evidence. “Many otherwise seemingly independent, university-based studies may be conducted for no other reason than to give physicians and researchers qualifications for promotion or tenure… Prestigious investigators may suppress via the peer review process the appearance and dissemination of findings that refute their findings, thus condemning their field to perpetuate false dogma,” wrote Ioannides in his 2005 PLoS article.

And finally, “The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.” Ioannides attributes this counter-intuitive effect to cutthroat competition among scientists to publish exciting research first. “This may explain why we occasionally see major excitement followed rapidly by severe disappointments in fields that draw wide attention,” he says. Isn’t this relevant to far-reaching claims made for embryonic stem cells?

Even more discouraging for medical researchers is that the gold-standard of medical research, double-blind randomised trials, are not altogether reliable either. In another 2005 paper published in the Journal of the American Medical Association, Ioannides examined 49 of the top science papers of the previous 13 years. They had appeared in the best journals and had been cited extensively. Yet between one-third and one-half of them were unreliable because they were later found to be either outright wrong or exaggerated.