text: Researchers ought to be able to duplicate the findings of other scientists’ work, but a new study suggests that many published psychology results can't be recreated. 
   
   A huge, collaborative research project attempted to recreate 100 studies that were recently published in major psychology journals, and it found that only 39 of those studies' results could be replicated. That could mean that the studies were wrong in the first place, but researchers say that the findings tell more about the difficulty of designing a reproducible study than the accuracy of the studies themselves. 
   
   Studies need to be reproducible so that scientists can confirm their effects. That's why scientists have generally pushed toward reproducing studies — and not just in psychology. In part, that's to catch scientific fraud, but it's also simply to make scientific findings more trustworthy. In January 2014, the National Institutes of Health announced it would create new initiatives to address these concerns, but there still aren't widely established reproducibility guidelines. The study being published today speaks to why a bigger focus on reproducibility is necessary. 
   
   "The results present an opportunity." 
   
   "I don’t see this story as negative or pessimistic," Brian Nosek, a University of Virginia researcher and the study's corresponding author, told reporters this week. "The results present an opportunity, and the project is a demonstration of science demonstrating one of its central qualities: self-correction." 
   
   That self-correction even extended to the 39 studies that researchers were able to replicate. Of those, 83 percent had smaller effects that were in the initial reports; the drop in effect size was usually around 50 percent. The report is being published today in the journal Science. 
   
   Nosek says that there are three reasons studies' results may not have been replicated or may have changed. First, it's possible that the original result identified an effect that wasn't real. Second, it's possible that the researchers failed to detect a real effect in the reproduction. Finally, it's possible that both studies were accurate but that their methodologies differed in slight but meaningful ways. 
   
   The scientists attempting to reproduce studies tried to hew closely to the originals; they even worked with the original researchers and published their study designs for feedback from the over 270 people who worked on the reproduction project. That certainly suggests that errors are being made throughout these experiments, which mostly focused on social psychology and cognitive psychology. But for these researchers, the differing results suggests something else: that there’s a problem with the reproduction process. 
   
   "Despite its central importance, there is little known about the reproducibility of research in general," Nosek said. "And there has been a growing concern that reproducibility may be lower than expected or desired." 
   
   Research is typically focused on new results, not reproduction 
   
   One factor encouraging that, Nosek suggests, is researchers' desire for publication. Succeeding in academia means regularly publishing work, and novel results — rather than reproductions of older results or results that fail to establish a correlation — are far more likely to make it into prestigious journals. "If this occurs on a broad scale, then the published literature may be more beautiful than reality," Nosek says. 
   
   To move toward more repeatable studies, researchers and journals need to shift their interests away from novelty, the current focus. That's part of why the people behind this study — operating as part of an initiative from the Open Science Collaboration called the Reproducibility Project: Psychology — worked with the original researchers whose studies they were replicating. It encouraged them to view these experiments as steps forward, not as threats to their work. 
   
   E.J. Masicampo, a researcher at Wake Forest University, both conducted one of the reproductions and had one of his own studies reproduced by someone else. "Mine was a study that did not successfully replicate," Masicampo told reporters. His study had looked at whether giving someone a sugary beverage made it easier for them to make mentally challenging decisions while fatigued. 
   
   Both Masicampo's study and its replication asked students to decide between apartments of different sizes and distances from a college campus. It turned out that the decision was difficult for students at Florida State, where Masicampo was working, but the question was too easy for students at the University of Virginia, where it was being reproduced. "What we found, I think, was that the [methodology] that worked at the original location ... didn't quite translate for participants at the replication location," Masicampo said. 
   
   How exact must a replication be? 
   
   The group was trying to be very faithful to the original study, Masicampo said. The fact that doing so resulted in a failed replication, "highlighted the issue of how exact these replications should be and how much things change going from one place to another." Were replication standards to be put in place, these questions might be answered and lead to a clearer view of each finding's accuracy. 
   
   The researchers want to underscore that this isn't a failure of science — it's just science working normally. "The real challenge here is that we so much want to have clear answers from any individual study, from any program of research, from any report about studies," Nosek said. "But science doesn't provide certainty. At least not immediately." 
   
   Of course, all that may make you wonder: how reproducible are these findings? "It's a very good question and I don't envy the group trying it again," Nosek said. "But it does illustrate the precise point of the reproducibility effort: that even this project itself is not a final word, a last word, a definitive word about reproducibility." ||||| Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. Scientific claims should not gain credence because of the status or authority of their originator but by the replicability of their supporting evidence. Even research of exemplary quality may have irreproducible empirical findings because of random or systematic error. 
   
   There is concern about the rate and predictors of reproducibility, but limited evidence. Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results. Direct replication is the attempt to recreate the conditions believed sufficient for obtaining a previously observed finding and is the means of establishing reproducibility of a finding with new data. We conducted a large-scale, collaborative effort to obtain an initial estimate of the reproducibility of psychological science. 
   
   We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. There is no single standard for evaluating replication success. Here, we evaluated reproducibility using significance and P values, effect sizes, subjective assessments of replication teams, and meta-analysis of effect sizes. The mean effect size (r) of the replication effects (M r = 0.197, SD = 0.257) was half the magnitude of the mean effect size of the original effects (M r = 0.403, SD = 0.188), representing a substantial decline. Ninety-seven percent of original studies had significant results (P < .05). Thirty-six percent of replications had significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams. 
   
   CONCLUSION 
   
   No single indicator sufficiently describes replication success, and the five indicators examined here are not the only ways to evaluate reproducibility. Nonetheless, collectively these results offer a clear conclusion: A large portion of replications produced weaker evidence for the original findings despite using materials provided by the original authors, review in advance for methodological fidelity, and high statistical power to detect the original effect sizes. Moreover, correlational evidence is consistent with the conclusion that variation in the strength of initial evidence (such as original P value) was more predictive of replication success than variation in the characteristics of the teams conducting the research (such as experience and expertise). The latter factors certainly can influence replication success, but they did not appear to do so here. 
   
   Reproducibility is not well understood because the incentives for individual scientists prioritize novelty over replication. Innovation is the engine of discovery and is vital for a productive, effective scientific enterprise. However, innovative ideas become old news fast. Journal reviewers and editors may dismiss a new test of a published idea as unoriginal. The claim that “we already know this” belies the uncertainty of scientific evidence. Innovation points out paths that are possible; replication points out paths that are likely; progress relies on both. Replication can increase certainty when findings are reproduced and promote innovation when they are not. This project provides accumulating evidence for many findings in psychological research and suggests that there is still more work to do to verify whether we know what we think we know.
target: – A new study on psychology research is essentially bashing all other psychology studies. The research area has gotten a bad rap recently thanks to retracted research papers, so psychologists set out to discover what was going on. When research is valid, others should be able to duplicate the study and come up with the same results. But when more than 250 scientists tried to re-create 100 studies from three respected psychology journals—working closely with original researchers—the results matched in just 39 cases. Even among those, 83% were found to have smaller effects than first reported, reports the Verge. The studies covered topics like personality, relationships, learning, and memory, and were the kind trusted by therapists and educators, reports the New York Times. A scientist argues the reproducibility issue could apply across other fields, too. "To see it so clearly, on such a large scale—it's unprecedented," adds another. "There is little known about the reproducibility of research in general," says study author Brian Nosek. "And there has been a growing concern that reproducibility may be lower than expected or desired." He adds that scientific journals tend to focus on new research, so those hoping to get published don't waste time testing older studies. Nosek doesn't consider the results "negative or pessimistic," but he hopes they'll lead to self-correction and replication standards. So are these results reproducible? "Even this project itself is not a final word, a last word, a definitive word about reproducibility," Nosek says.