District-Level Structural Heterogeneity in Experimental Compliance
RCTClinical Trial
Multi-site field experiments in public school systems face site-specific administrative and certification constraints that cause treatment compliance rates to vary widely across participating jurisdictions.
Picture this
Running an identical clinical trial across six different hospital branches where certain locations have mandatory patient reassignment rules will cause treatment retention to vary wildly by site, regardless of the central experimental protocol.
What the evidence says
First-stage compliance coefficients ranged from 0.744 in Dallas and 0.517 in Charlotte-Mecklenburg down to 0.157 in New York and -0.079 (non-significant) in Denver. Low compliance in Denver stemmed from structural language certification rules requiring Spanish-speaking students to remain with Spanish-certified teachers.
- Who was studied
- N = 31,439 randomized students across 6 urban public school districts (Dallas, Charlotte-Mecklenburg, Hillsborough, Memphis, New York City, and Denver).
- How
- District-by-district first-stage instrumental variable regression of actual teacher effectiveness on assigned teacher effectiveness.
What to do
Audit local institutional constraints and allow first-stage compliance rates to vary by site when evaluating multi-district field experiments.
From the source
"Due to higher levels of compliance with the randomly assigned rosters, the effectiveness of the assigned teachers were more likely to filter through to actual teacher effectiveness in Dallas and Charlotte-Mecklenburg (.744 and .517, respectively) than in Hillsborough, Memphis, and New York (.380, .277, and .157, respectively)."
Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment