Between-School Generalizability Limit of Within-School RCTs
RCTClinical Trial
Randomized field experiments that restrict student and teacher reassignments to within-school blocks cannot evaluate whether teacher performance measures reliably compare instructor quality across different school sites.
Picture this
Evaluating two runners competing on the same track provides a fair comparison of their relative speed, but that match cannot prove how a runner on a smooth track compares to a runner on an uphill mountain trail. Without swapping runners across tracks, cross-site comparisons remain unvalidated.
What the evidence says
Random assignment was strictly bounded within individual schools, grade levels, and subjects, preventing statistical evaluation of whether teacher value-added metrics hold predictive validity for cross-school teacher transfers or district-wide comparisons.
- Who was studied
- N = 1,181 randomized teachers in 284 urban public schools across 6 school districts.
- How
- Blocked randomized controlled trial restricting random roster assignments strictly to exchangeable teacher groups within individual school buildings.
What to do
Limit policy claims regarding teacher value-added validity to within-school comparisons unless inter-school random reassignment data become available.
From the source
"Because we could not randomly assign students or teachers to a different school site, our study does not allow us to investigate the validity of the measures of effectiveness for gauging differences across schools."
Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment