Thursday, September 8, 2011

Is Science Broken? Part 2: Is Most Academic Research Unreliable?

The answer appears to be "Yes", at least in the academic setting:

"Bayer halts nearly two-thirds of its target-validation projects because in-house experimental findings fail to match up with published literature claims, finds a first-of-a-kind analysis on data irreproducibility.

"An unspoken industry rule alleges that at least 50% of published studies from academic laboratories cannot be repeated in an industrial setting, wrote venture capitalist Bruce Booth in a recent blog post. A first-of-a-kind analysis of Bayer’s internal efforts to validate ‘new drug target’ claims now not only supports this view but suggests that 50% may be an underestimate; the company’s in-house experimental data do not match literature claims in 65% of target-validation projects, leading to project discontinuation.

"“People take for granted what they see published,” says John Ioannidis, an expert on data reproducibility at Stanford University School of Medicine in California, USA. “But this and other studies are raising deep questions about whether we can really believe the literature, or whether we have to go back and do everything on our own.”"

So the academics are just faking stuff wholesale to advance their careers?  This is pretty shocking, but not surprising.  Stuff like this has been going on in the liberal arts, which I studied, to the point that I'd argue pursuing a liberal arts degree in 99% of schools is probably a waste of time.

From the comments in one of the links from the page above:

"This article highlights an essential difference between how success is defined in academia vs. industry. Academics make their names with high profile papers in journals like Nature, whose selection process favors the new and exciting 'big ideas' from individuals deemed to be rising stars by their peers. In general, unless overt fraud can be demonstrated, it makes no difference to the individual's career if their high profile papers are later found to be incorrect. However, in industry, the bottom line for success is whether the research will lead to a new product, so there is little incentive for researchers to overhype their data since they will be accountable if it is not reproducible."

That about nails it.

Part 1.