November 10, 2015

COME BACK, ADMIRAL POINDEXTER, ALL IS FORGIVEN:

How To Tell Good Studies From Bad? Bet On Them (CHRISTIE ASCHWANDEN, 11/09/15, 538)

Although replication is essential for verifying results, the current scientific culture does little to encourage it in most fields. That's a problem because it means that misleading scientific results, like those from the "shades of gray" study, could be common in the scientific literature. Indeed, a 2005 study claimed that most published research findings are false.

There's a growing recognition of the problem, but replication takes time and money, and with funding for science at a premium, there's an urgent need to prioritize.

Today in the Proceedings of the National Academy of Sciences, Nosek and an international team of researchers present a tool for doing that -- betting. They found that compared to simply asking experts to predict the likelihood that studies will be reproduced, asking them to bet money on the outcomes improved the accuracy of the guesses.

The researchers began by selecting some studies slated for replication in the Reproducibility Project: Psychology -- a project that aimed to reproduce 100 studies published in three high-profile psychology journals in 2008. They then recruited psychology researchers to take part in two prediction markets. These are the same types of markets that people use to bet on who's going to be president. In this case, though, researchers were betting on whether a study would replicate or not.

Before each prediction market began, participants (47 actively took part in the first market, 45 traded in the second) were asked two questions: How likely do you think it is that each hypothesis in this market will be replicated, and how well do you know this topic?

They were then given points worth a total of $100 to bet on whether the studies in their prediction market would replicate. A replication was considered successful if it produced a result, with a p-value of less than 0.05, in the same direction as the original result. Players entered the market with 10,000 points each and could buy and sell contracts for each hypothesis. If a replication succeeded, then its share paid 100, but if the replication failed, then it paid nothing. "If you believe the result will be replicated, you buy the contract, which increases the price," said the study's lead author, Anna Dreber, an economist at the Stockholm School of Economics. "If you don't believe in a study, then you can short-sell it."

A study's final share price when the market closed was akin to an estimated probability that the study would replicate successfully, Nosek said. "A price of 75 indicates that the market perceived a 75 percent likelihood of replication success," he said.

The prediction market correctly called nearly three-quarters (71 percent) of the attempted replications, 39 percent of which succeeded in the reproducibility project.



MORE:
Pentagon Prepares A Futures Market On Terror Attacks (CARL HULSE, July 29, 2003, NY Times)

The Pentagon office that proposed spying electronically on Americans to monitor potential terrorists has a new experiment. It is an online futures trading market, disclosed today by critics, in which anonymous speculators would bet on forecasting terrorist attacks, assassinations and coups.

Traders bullish on a biological attack on Israel or bearish on the chances of a North Korean missile strike would have the opportunity to bet on the likelihood of such events on a new Internet site established by the Defense Advanced Research Projects Agency.

The Pentagon called its latest idea a new way of predicting events and part of its search for the ''broadest possible set of new ways to prevent terrorist attacks.'' Two Democratic senators who reported the plan called it morally repugnant and grotesque. The senators said the program fell under the control of Adm. John M. Poindexter, President Ronald Reagan's national security adviser.

Posted by at November 10, 2015 5:09 PM

  

« WOMEN'S LIBERATION STRIKES AGAIN: | Main | THE TORTOISE AND THE HIRE: »