... studies with low P values are thought to be convincing, and so are not often repeated to be sure the results are correct. This might seem reasonable because there is limited money and time in science – results from a study that seem very clear perhaps do not warrant double-checking when there are new discoveries out there to be made.

<snip>

This weakness could well explain why famous scientific findings from the past, central to the foundations of many disciplines, are not being confirmed now that the original studies are finally being re-examined.

These include a lack of reproducibility in cancer research, ...

<snip>

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3917235/

Negativity towards negative results: a discussion of the disconnect between scientific worth and scientific culture

Natalie Matosin,1,2,* Elisabeth Frank,1,2 Martin Engel,1,2 Jeremy S. Lum,1,2 and Kelly A. Newell1,2

Abstract

Science is often romanticised as a flawless system of knowledge building, where scientists work together to systematically find answers. In reality, this is not always the case. Dissemination of results are straightforward when the findings are positive, but what happens when you obtain results that support the null hypothesis, or do not fit with the current scientific thinking? In this Editorial, we discuss the issues surrounding publication bias and the difficulty in communicating negative results. Negative findings are a valuable component of the scientific literature because they force us to critically evaluate and validate our current thinking, and fundamentally move us towards unabridged science.

“What gets us into trouble is not what we don’t know, it’s what we know for sure that just ain’t so.” – Mark Twain.

The impact of negative findings

Increasingly, there is pressure on scientists to choose investigative avenues that result in high-impact knowledge. This challenge has, in many cases, swayed scientists to pursue paths of investigation that are not necessarily logical or hypothesis-driven. Rather than approaching a research question in a systematic manner, it seems that scientists are encouraged to pursue non-linear lines of investigation in search of significance, and many that have the luxury are known to tuck away negative findings (the ‘file-drawer’ effect) and focus on their positive outcomes (Scargle, 1999). This behaviour likely stems from an ever-heightening hurdle that scientists need to jump: high publication output with a high citation rate in order to win competitive grants to drive their research, move up the rungs and pay the bills.

Published a few years ago in PLoS ONE, Daniele Fanelli states that “Papers are less likely to be published and to be cited if they report ‘negative’ results” (Fanelli, 2010). Because scientists are involuntarily finding themselves engaged in competition for positions and funding, many are choosing not to proceed with their non-significant findings (those that support the null hypothesis) that yield less scientific interest and fewer citations. Consequently, the amount of non-significant data reported is progressively declining (Fanelli, 2012). Although it could be argued that this is due to an increasing quality of science, it is more likely attributable to the selectiveness of ‘high impact’ journals that, in our opinion, might as well have a bold statement in the submission form: negative results are not accepted. However, there seems to be a gap between results that are positive and results that are high impact. Logically there is no connection, but it seems scientific culture assumes that they are analogous. Why aren’t negative results considered to be of the same value?

<snip>

Examples from psychiatric research

At a recent conference, two colleagues discovered that they had both unsuccessfully attempted to alter depression-like behaviour in the CD1 mouse strain (a widely used animal model for toxicology studies) with a variety of classical antipsychotics. These findings were surprising in light of the many studies demonstrating efficacy of antipsychotic drugs in different experimental models. Realising that this had occurred in two separate labs, they considered that this might not have been a lack of experiments performed using CD1 mice, but rather a lack of publications on negative findings. They have since corroborated their findings with others. Because the results were unpublished, research groups had continued to follow the same lines of thought and the same paths of investigation, only to all fail in the same way, ultimately wasting time and resources. To our knowledge, these results remain unpublished.

<snip>

http://www.nature.com/nmeth/journal/v12/n3/full/nmeth.3288.html

The fickle P value generates irreproducible results

Lewis G Halsey, Douglas Curran-Everett, Sarah L Vowler & Gordon B Drummond

Nature Methods 12, 179–185 (2015)
doi:10.1038/nmeth.3288

Published online
26 February 2015

The reliability and reproducibility of science are under scrutiny. However, a major cause of this lack of repeatability is not being considered: the wide sample-to-sample variability in the P value. We explain why P is fickle to discourage the ill-informed practice of interpreting analyses based predominantly on this statistic.

<snip>

...

The study used text mining to extract p-values - a number that indicates how likely it is that a result occurs by chance - from more than 100,000 research papers published around the world, spanning many scientific disciplines, including medicine, biology and psychology.

"Many researchers are not aware that certain methods could make some results seem more important than they are. They are just genuinely excited about finding something new and interesting," Dr Head said.

"I think that pressure to publish is one factor driving this bias. As scientists we are judged by how many publications we have and the quality of the scientific journals they go in.

...

Dr Head said the study found a high number of p-values that were only just over the traditional threshold that most scientists call statistically significant.

...