The legitimacy of scientific experiments and studies has always hinged on one thing: reproducibility. If a finding does not recur in another study, it could have been a fluke, or an otherwise unreliable piece of information.

A recent review study suggests that replicating findings isn’t so easy — an examination of findings across multiple criteria showed that researchers could replicate less than half of their original findings. The findings could contest the validity of certain scientific claims, but it also points to the difficulty of conducting effective replications and achieving reproducible results.

The study, called the Reproducibility Project: Psychology, was the largest and most comprehensive investigation about the rate and predictors of reproducibility in science and was conducted by over 270 researchers on five continents.

"For years there has been concern about the reproducibility of scientific findings, but little direct, systematic evidence. This project is the first of its kind and adds substantial evidence that the concerns are real and addressable," said Brian Nosek, a U.Va. Psychology professor and coordinator of the study, in a press release.

The study identified some evidence of certain influences on reproducibility, according to Anup Gampa, a Reproducibility Project team member and Ph.D. candidate at U.Va. He said that science is a unique way to gather knowledge, since it relies on reproducibility to gain confidence in evidence and ideas.

Elizabeth Gilbert, a team member and Ph.D. candidate at U. Va, said that a failure to reproduce does not necessarily mean the report was incorrect.

"A replication team must have a complete understanding of the methodology used for the original research, and shifts in the context or conditions of the research could be unrecognized but important for observing the result," she said.

Nosek noted that a problem for psychology in particular, along with other disciplines, is that scientists’ incentives are not dependably aligned with reproducibility.

"Scientists aim to contribute reliable knowledge but also need to produce results that help them keep their job as a researcher," he said. "To thrive in science, researchers need to earn publications, and some kinds of results are easier to publish than others, particularly ones that are novel and show unexpected or exciting new directions."

According to Nosek and his colleagues, scientists pursue innovative research in the best interests of their careers, even if it costs them the reproducibility of findings. Research that produces new or surprising findings is more likely to be published than research examining reproducibility, according to the authors.

As for the low levels of reproducibility found in the project, the team suggested three main reasons for the occurrence — small differences in when, where, or how the replication was carried out; the replication could have failed, by chance, to detect the original result; and the original result could have been a false positive.

"The findings demonstrate that reproducing original results may be more difficult than is presently assumed, and interventions may be needed to improve reproducibility," said Johanna Cohoon, a project coordinator with the Charlottesville-based Center for Open Science.

Several organizations are working to improve reproducibility, including the journal Psychological Science.

"Efforts include increasing transparency of original research materials, code and data so that other teams can more accurately assess, replicate and extend the original research, and pre-registration of research designs to increase the robustness of the inferences drawn from the statistical analyses applied to research results," said Denny Borsboom, a project team member from the University of Amsterdam.

Source: Nosek B, et al. Promoting an open research culture. Science. 2015.