When a famous study fails to replicate, was the effect never real, or does it depend on context?

Scientists disagree, and three main explanations compete. The first is context. Jay Van Bavel and colleagues argue that failures to replicate may be explained by contextual differences between the original experiment and the replication, often called "hidden moderators". When they rated the effects in the Reproducibility Project by how sensitive to context they seemed, higher context sensitivity went with a lower chance of replicating, even after accounting for things like the original's sample size and the replication's power. The 60-lab project described in the replication-crisis lesson cuts the other way: when its findings failed, they failed consistently across contexts.

The second is heterogeneity: an effect may genuinely come in several sizes rather than one. In a replication across 36 sites, 8 of 16 effects varied significantly between sites, yet the deliberate differences between those sites explained very little of the variation. Across 200 meta-analyses of psychological effects, the typical variation was what the authors call "huge", enough that matching the original's effect size is unlikely even with very large replications.

The third is that the first result was weak or wrong. In the Reproducibility Project, only 25 percent of the non-replications directly contradicted the original, while 49 percent were inconclusive because of underpowered designs. And the philosopher Alexander Bird adds a base-rate argument: if only 10 percent of tested hypotheses are true, as many as 36 percent of results will be false positives, so low replication rates could be consistent with quality science.