Between-School Generalizability Limit of Within-School RCTs

In Simple Words

Evaluating two runners competing on the same track provides a fair comparison of their relative speed, but that match cannot prove how a runner on a smooth track compares to a runner on an uphill mountain trail. Without swapping runners across tracks, cross-site comparisons remain unvalidated.

Research question

Randomized field experiments that restrict student and teacher reassignments to within-school blocks cannot evaluate whether teacher performance measures reliably compare instructor quality across different school sites.

Participants

N = 1,181 randomized teachers in 284 urban public schools across 6 school districts.

Study Design

Blocked randomized controlled trial restricting random roster assignments strictly to exchangeable teacher groups within individual school buildings.

Results

Random assignment was strictly bounded within individual schools, grade levels, and subjects, preventing statistical evaluation of whether teacher value-added metrics hold predictive validity for cross-school teacher transfers or district-wide comparisons.

Takeaway

Limit policy claims regarding teacher value-added validity to within-school comparisons unless inter-school random reassignment data become available.

Read alongside the research

Plain-English glossary

Original explanations, informed by the references below. Examples are illustrations, not data from the studies above. The research text stays unchanged.

Randomized controlled trial

A study that uses chance, like drawing lots, to decide who receives each option being compared. Researchers compare the groups’ outcomes to estimate the effect of those options.

ExampleA lottery assigns volunteers to a training programme or a comparison group. Researchers then compare their outcomes.

GET-IT glossary · Randomized study ↗Read more about randomized controlled trial →

Related findings

• Intent-to-Treat (ITT) Reduced-Form Effect Estimation →Education

  If a doctor assigns patients to a specific health routine but some patients do not follow it, checking how much healthier the whole assigned group became measures the effectiveness of giving out the advice, rather than the physical impact of doing the routine itself.

• Within-School Exchangeable Rosters and Randomization Blocks →Education

  Instead of shuffling players across an entire sports league where positions and rules differ, organizers only swap players between teams that play in the same division during the exact same match time, ensuring fair substitutes without breaking the game schedule.

• Academic Grade-Incentivized Curriculum Compliance →Education

  Linking software progress to a real school report card acts like a mandatory gate on a turnstile. Students must complete digital quizzes and modules because their software performance directly impacts their overall grade in an established class like computer science.

• Academic Remediation Inefficacy in Postsecondary Persistence →Education

  Standard developmental remediation is like forcing aspiring drivers to complete months of classroom parking simulations before letting them drive on real roads. Because the simulations take time and money without moving drivers closer to their destination, most drivers become discouraged and quit driving before ever reaching the main highway.