In Simple Words
Separating a radio broadcast from static by comparing recordings from two different days; persistent sounds that appear on both days represent the true broadcast signal, while transient pops represent background noise.
Observed teacher score records
- Math
- Persistent spread: 0.16–0.18
- English
- Persistent spread: 0.10–0.12
2008–10 records across teacher samples. Spread is in student standard deviations; no treatment comparison.
See source →Research question
Single-year test score variance overstates true teacher effectiveness differences because raw year-to-year variation includes transient student sampling noise and classroom-level sorting volatility.
Participants
N = 1,181 randomized teachers, 3,802 non-randomized school teachers, and 17,153 non-study district teachers across grades 4–8 in 6 urban school districts.
Study Design
Cross-year covariance estimation of value-added scores between 2008–09 and 2009–10 to isolate persistent signal variance from temporary noise.
Results
The estimated signal standard deviation of true teacher effects ranged from 0.16 to 0.18 student-level standard deviations in math and 0.10 to 0.12 in English Language Arts (ELA). ELA demonstrated lower signal variance and higher year-to-year volatility due to reliance on multiple-choice reading comprehension items without direct writing assessments.
Takeaway
Calculate cross-year covariance of performance metrics to isolate true signal standard deviation before establishing performance benchmark thresholds.