Cross-Sample Intervention Replicability
Pilot educational interventions frequently exhibit inflated effect sizes that fail to replicate when re-tested across independent cohorts, creating uncertainty for large-scale policy adoption.
Picture this
Imagine testing a new training routine on one sports team and seeing players run faster; to prove the routine actually works and wasn't just luck with that specific team, a coach applies the exact same routine to a second, completely separate team in a different league and verifies that their speed improves by the same margin.
What the evidence says
Key treatment effects replicated across both cohorts with striking consistency: persistent choice of challenging tasks increased by 8.5 percentage points in Sample A (p < 0.01) and 12.9 percentage points in Sample B (p < 0.01); week-ahead commitment increased by 14.3 percentage points in Sample A (p < 0.01) and 19.1 percentage points in Sample B (p < 0.01); and second-visit success increased by 8.7 percentage points in Sample A (p < 0.01) and 10.7 percentage points in Sample B (p < 0.05).
- Who
- N = 2,900 elementary school students across two independent cohorts (Sample A: N = 1,717 in 36 schools; Sample B: N = 1,203 in 16 schools) in Istanbul, Turkey.
- How
- Sequential randomized controlled field trials conducted across two distinct academic years (2013–2014 and 2015–2016) using identical curricular materials and teacher training protocols.
What to do
Validate social policy interventions by executing an independent replication RCT on a secondary cohort prior to authorizing full-scale institutional rollout.
From the source
"We find that the estimated treatment effects are remarkably similar across the two independent samples in which we randomize the educational intervention. The replicability of our results is encouraging and clears the path for a potential scale-up."
Ever Failed, Try Again, Succeed Better: Results from a Randomized Educational Intervention on Grit