PSYCHOLOGY
100 studies, repeated
A 2015 project re-ran 100 psychology studies. 36% of the replications had significant results, against 97% of the originals.
See what held upWHAT THE RESEARCH SAYS
Once is a finding.
Twice is evidence.
A single study is a first draft, not the final word.
See what held up
97% → 36%significant, then repeated
- One study is a start
- Effects often shrink
- Plans before data help
- Trust grows with repeats
Replication means running a study again to see if the result holds. When researchers did this at scale, many results held up only partly, and repeated effects were often about half as big. Science has been changing how it works in response.
Updated
THE SHORT ANSWER
Many results hold,
many come out smaller.
Three things large replication projects
and reform efforts have shown.
TOP JOURNALS
Nature and Science too
Of 21 social science experiments from Nature and Science, 13 replicated, with effects on average about half the original size.
See the numbers
THE FIX
Review the plan first
In a Registered Report, the methods are peer reviewed before the data are collected, so publication does not depend on the result.
How to read a studyA QUICK RESULTS CHECK
How much
held up?
Pick a measure from two large replication projects to see the result. The bar shows roughly how much of the original evidence held up on that measure.
Different measures answer different questions, which is why the figures vary. A failed replication does not by itself prove an original result was wrong.
TWO NUMBERS WORTH KNOWING
Smaller effects,
bigger samples.
One figure on what replications find,
one on how publishing is changing.
≈ 50%
Replication effect as a share of the original
Repeating 21 experiments from Nature and Science with samples about five times larger, researchers found effects on average about 50% of the original size.
EFFECT SIZE · NATURE HUMAN BEHAVIOUR
300+
Journals using the Registered Reports format
The Center for Open Science says over 300 journals now use Registered Reports, where peer review happens before data collection.
REGISTERED REPORTS · COS
The 2018 study also found that researchers' bets on which results would replicate were strongly related to what actually replicated, suggesting failures were not down to chance alone.
FOUR QUESTIONS FOR A HEADLINE
Read one study
like a scientist.
None of these needs a statistics degree. They are the questions replication researchers ask before trusting a single result.
Has it been repeated?
A finding confirmed by an independent team is stronger than one result, however striking.
How big was the study?
The 2018 replications used samples about five times larger than the originals, and the effects shrank.
Was the plan set in advance?
Registered Reports review the methods before the data arrive. As one editor puts it, the incentive shifts from the most beautiful story to the most accurate one.
How strong was the evidence?
The 2015 project found replication success was better predicted by the strength of the original evidence than by who ran the studies.
IN THE UK
Fixing the system,
not just the studies.
The UK Reproducibility Network is a national peer-led consortium that studies what makes research robust, runs training and shares good practice. It works across disciplines, from the arts and humanities to the physical sciences, with a particular focus on biomedical research.
Its free primers cover practices such as data sharing, preprints, open code and pre-registration with Registered Reports, each explaining what the practice is, why it helps and how to do it.
Questions
Questions people ask
What does replication mean in science?
It means running a study again, ideally with the original materials and a larger sample, to see whether the same result appears. The 2015 psychology project used high-powered designs and original materials where available.
How many psychology studies replicate?
In the 2015 Open Science Collaboration project, 36% of 100 replications had significant results, 39% were rated as replicating, and effects were half the size on average. Different measures give different answers.
Does a failed replication mean the original was fraud?
No. The 2018 authors estimate that both false positives and inflated effect sizes in genuine findings contribute to imperfect reproducibility. They also found that failures to replicate were not the result of chance alone.
What is a Registered Report?
A publishing format in which peer review happens before data are collected. Strong study plans are provisionally accepted, so publication does not depend on whether the result is positive or negative.
Is this only a problem in psychology?
No. The UK Reproducibility Network says these issues affect all disciplines, which is why it works across fields, with a particular focus on the biomedical sciences.
Where the facts come from
- Open Science Collaboration. Estimating the reproducibility of psychological science. Science 2015
- Camerer CF et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour 2018
- Center for Open Science: Registered Reports
- UK Reproducibility Network: About UKRN
- UK Reproducibility Network: Open Research Primers