Welcome to DU! The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards. Join the community: Create a free account Support DU (and get rid of ads!): Become a Star Member Latest Breaking News General Discussion The DU Lounge All Forums Issue Forums Culture Forums Alliance Forums Region Forums Support Forums Help & Search

jakeXT

(10,575 posts)
Fri Jun 19, 2015, 03:44 AM Jun 2015

Believe It Or Not, Most Published Research Findings Are Probably False

The rise of the Internet has worked wonders for the public's access to science, but this has come with the side effect of a toxic combination of confirmation bias and Google, enabling us to easily find a study to support whatever it is that we already believe, without bothering to so much as look at research that might challenge our position — or the research that supports our position for that matter. I'm certainly not immune myself from credulously accepting research that has later been called into question, even on this blog where I take great effort to take a skeptical approach and highlight false claims arising from research. Could it be the case that studies with incorrect findings are not just rare anomalies, but are actually representative of the majority of published research?

The claim that "most published research findings are false" is something you might reasonably expect to come out of the mouth of the most deluded kind of tin-foil-hat-wearing-conspiracy-theorist. Indeed, this is a statement oft-used by fans of pseudoscience who take the claim at face value, without applying the principles behind it to their own evidence. It is however, a concept that is actually increasingly well understood by scientists. It is the title of a paper written 10 years ago by the legendary Stanford epidemiologist John Ioannidis. The paper, which has become the most widely cited paper ever published in the journal PLoS Medicine, examined how issues currently ingrained in the scientific process combined with the way we currently interpret statistical significance, means that at present, most published findings are likely to be incorrect.

Richard Horton, the editor of The Lancet recently put it only slightly more mildly: "Much of the scientific literature, perhaps half, may simply be untrue." Horton agrees with Ioannidis' reasoning, blaming: "small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance." Horton laments: "Science has taken a turn towards darkness."

Last year UCL pharmacologist and statistician David Colquhoun published a report in the Royal Society's Open Science in which he backed up Ioannidis' case: "If you use p=0.05 to suggest that you have made a discovery, you will be wrong at least 30 percent of the time." That's assuming "the most optimistic view possible" in which every experiment is perfectly designed, with perfectly random allocation, zero bias, no multiple comparisons and publication of all negative findings. Colquhorn concludes: "If, as is often the case, experiments are underpowered, you will be wrong most of the time."

http://bigthink.com/neurobonkers/believe-it-or-not-most-published-research-findings-are-probably-false

6 replies = new reply since forum marked as read
Highlight: NoneDon't highlight anything 5 newestHighlight 5 most recent replies
Believe It Or Not, Most Published Research Findings Are Probably False (Original Post) jakeXT Jun 2015 OP
We need to do some research on this! Human101948 Jun 2015 #1
Good research is peer reviewed before publication precisely to avoid the publication JDPriestly Jun 2015 #2
Might be true of *medical* research reports ... eppur_se_muova Jun 2015 #3
Probably understates the error rate for education research. Igel Jun 2015 #4
Well, yeah ... or any area of sociology ... eppur_se_muova Jun 2015 #5
I think things are getting better... Sancho Jun 2015 #6
 

Human101948

(3,457 posts)
1. We need to do some research on this!
Fri Jun 19, 2015, 05:46 AM
Jun 2015

Kind of ballpark estimates, aren't they? "Much of the scientific literature, perhaps half, may simply be untrue..." Well that isn't "most" is it? "Perhaps?" "May be?" This is redolent of B.S.

JDPriestly

(57,936 posts)
2. Good research is peer reviewed before publication precisely to avoid the publication
Fri Jun 19, 2015, 06:48 AM
Jun 2015

of "false" research. Still, science is about learning new things. So of course, it is not a stagnant body of knowledge.

The point in science is constantly learning.

Science constantly challenges our assumptions.

The Bible doesn't change over time. Some scholars picked some manuscripts, put them together into a book in a certain order, and that's that. A lot of people want science to be that kind of black and white, true or false certainty.

But science is the opposite. It is a developing understanding of the universe around us.

So??? What's this thread really about?

eppur_se_muova

(36,275 posts)
3. Might be true of *medical* research reports ...
Fri Jun 19, 2015, 11:38 AM
Jun 2015

where one is examining complex organisms with their own contigent histories and must rely on extensive statistical analysis to extract any interpretable data at all.

Most scientific research, particularly in the physical sciences, is on much firmer footing, if for no other reason than that it is much simpler to attempt to reproduce experimental results. Even in fields that are almost purely observational, rather than experimental, the connections between cause and effect are much less tangled, thus more easily observed and more easily verified.

The authors of this piece really should be taken to task for failing to make such important distinctions. The "published research reports" they are referring to are medical studies carried out using the tools and methods of science, not research in science itself. Medicine is still more art than science, and always will be, because every patient is uniquely different from every one previously studied.

Igel

(35,332 posts)
4. Probably understates the error rate for education research.
Fri Jun 19, 2015, 03:17 PM
Jun 2015

In which one is examining complex organisms interacting in a complex system, controls are rare, methodology is often sloppy, changing just one thing at a time is well nigh impossible, the thinking highly "motivated" because we must "think of the children!", and the statistical analysis is formulaic without pondering degrees of freedom and reasonable default hypothesis.

In one metastudy, a researcher and his "lab" looked over 300+ ed research articles on a given subfield. Between misquoted definitions, lack of controls, changing more variables than they were accounting for, and just bad data collection and analysis they found 2 studies that passed their quality check. One found a barely statistically significant positive effect. The other found a barely statistical negative effect.

eppur_se_muova

(36,275 posts)
5. Well, yeah ... or any area of sociology ...
Fri Jun 19, 2015, 04:57 PM
Jun 2015

where people can't agree on basic definitions, and are basically biased because the object of the study is people, and the researchers are themselves people, which makes true objectivity impossible.

Sancho

(9,070 posts)
6. I think things are getting better...
Sat Jun 20, 2015, 01:13 PM
Jun 2015

There have been complaints about the quality of research designs, hypothesis testing, etc. for many decades. At least today, we have much more access to international research, and large databases of information.

Even though there are certainly issues with funding biased research, publish-if-you-pay, and reviewers who make mistakes; overall the good quality publications are excellent. It's hard to even keep up in some narrow fields.

I'd say the big issue to me is insisting that researchers get grants or produce patents to "make money". This pushes science to the applied side, so good basic ideas may not get attention.

For college faculty, the reliance on "citation indexes" for tenure has also led to researchers playing games for survival, and that may eventually affect research reporting. "Open" journals online is also changing the game.

We live in interesting times!

Latest Discussions»Culture Forums»Science»Believe It Or Not, Most P...