District-Level Structural Heterogeneity in Experimental Compliance

In Simple Words

Running an identical clinical trial across six different hospital branches where certain locations have mandatory patient reassignment rules will cause treatment retention to vary wildly by site, regardless of the central experimental protocol.

Research question

Multi-site field experiments in public school systems face site-specific administrative and certification constraints that cause treatment compliance rates to vary widely across participating jurisdictions.

Participants

N = 31,439 randomized students across 6 urban public school districts (Dallas, Charlotte-Mecklenburg, Hillsborough, Memphis, New York City, and Denver).

Study Design

District-by-district first-stage instrumental variable regression of actual teacher effectiveness on assigned teacher effectiveness.

Results

First-stage compliance coefficients ranged from 0.744 in Dallas and 0.517 in Charlotte-Mecklenburg down to 0.157 in New York and -0.079 (non-significant) in Denver. Low compliance in Denver stemmed from structural language certification rules requiring Spanish-speaking students to remain with Spanish-certified teachers.

Takeaway

Audit local institutional constraints and allow first-stage compliance rates to vary by site when evaluating multi-district field experiments.

Related findings

• Academic Grade-Incentivized Curriculum Compliance →Education

  Linking software progress to a real school report card acts like a mandatory gate on a turnstile. Students must complete digital quizzes and modules because their software performance directly impacts their overall grade in an established class like computer science.

• Academic Remediation Inefficacy in Postsecondary Persistence →Education

  Standard developmental remediation is like forcing aspiring drivers to complete months of classroom parking simulations before letting them drive on real roads. Because the simulations take time and money without moving drivers closer to their destination, most drivers become discouraged and quit driving before ever reaching the main highway.

• Administrative Algorithmic Dropout Targeting →Education

  Instead of waiting for students to start failing classes, administrators analyze prior middle school report cards and standardized admission test scores before the academic year begins to calculate an individual risk score. This functions like a health screening questionnaire at a hospital lobby that flags patients with elevated blood pressure before an acute cardiac event takes place.

• Advisor-Financial Role Separation →Education

  Separating counseling from financial auditing is like having an attorney represent a client in court while a separate bank manager evaluates the client's loan application. Because the attorney does not decide on loan approvals, the client can openly share personal struggles with their lawyer without fearing financial penalty.