Cross-Sample Intervention Replicability

In Simple Words

Imagine testing a new training routine on one sports team and seeing players run faster; to prove the routine actually works and wasn't just luck with that specific team, a coach applies the exact same routine to a second, completely separate team in a different league and verifies that their speed improves by the same margin.

Research question

Pilot educational interventions frequently exhibit inflated effect sizes that fail to replicate when re-tested across independent cohorts, creating uncertainty for large-scale policy adoption.

Participants

N = 2,900 elementary school students across two independent cohorts (Sample A: N = 1,717 in 36 schools; Sample B: N = 1,203 in 16 schools) in Istanbul, Turkey.

Study Design

Sequential randomized controlled field trials conducted across two distinct academic years (2013–2014 and 2015–2016) using identical curricular materials and teacher training protocols.

Results

Key treatment effects replicated across both cohorts with striking consistency: persistent choice of challenging tasks increased by 8.5 percentage points in Sample A (p < 0.01) and 12.9 percentage points in Sample B (p < 0.01); week-ahead commitment increased by 14.3 percentage points in Sample A (p < 0.01) and 19.1 percentage points in Sample B (p < 0.01); and second-visit success increased by 8.7 percentage points in Sample A (p < 0.01) and 10.7 percentage points in Sample B (p < 0.05).

Takeaway

Validate social policy interventions by executing an independent replication RCT on a secondary cohort prior to authorizing full-scale institutional rollout.

Read alongside the research

Plain-English glossary

Original explanations, informed by the references below. Examples are illustrations, not data from the studies above. The research text stays unchanged.

Percentage points

The difference between two percentages.

ExampleA rise from 40% to 50% is 10 percentage points. Relative to the starting value, that is a 25% increase.

Office for National Statistics · Percentages ↗Read more about percentage points →

Randomized controlled trial

A study that uses chance, like drawing lots, to decide who receives each option being compared. Researchers compare the groups’ outcomes to estimate the effect of those options.

ExampleA lottery assigns volunteers to a training programme or a comparison group. Researchers then compare their outcomes.

GET-IT glossary · Randomized study ↗Read more about randomized controlled trial →

Related findings

• Active Tracking Attrition Mitigation →Methods & evidence

  Dispatching field teams directly to children's homes when they miss school testing operates like a home delivery service tracking down uncollected mail to ensure every package is accounted for.

• Experimental Self-Selection Verification and External Validity →Methods & evidence

  Think of a free health clinic offering checkups to local residents. To prove that clinic attendees represent the whole town rather than just the sickest or richest residents, doctors compare the age, income, and health history of attendees against census data for the entire town.

• Shift-Isolated Educational Unit Partitioning →Methods & evidence

  When two completely different groups of students use the exact same school building at different times of day—one group leaving before the second group arrives—the physical building is shared, but the social environments are totally separated by time, functioning like two completely separate schools under one roof.

• Two-Wave Strategic Baseline Suspension →Methods & evidence

  Imagine starting a public street survey and realizing people are hesitant to talk to strangers. You temporarily stop surveying, form an official partnership with a major trusted local bank, and reopen your survey desks right inside the bank's lobby branches where customers already feel secure.