Signal Variance Decomposition of Teacher Effects
RCTClinical Trial
Single-year test score variance overstates true teacher effectiveness differences because raw year-to-year variation includes transient student sampling noise and classroom-level sorting volatility.
Picture this
Separating a radio broadcast from static by comparing recordings from two different days; persistent sounds that appear on both days represent the true broadcast signal, while transient pops represent background noise.
What the evidence says
The estimated signal standard deviation of true teacher effects ranged from 0.16 to 0.18 student-level standard deviations in math and 0.10 to 0.12 in English Language Arts (ELA). ELA demonstrated lower signal variance and higher year-to-year volatility due to reliance on multiple-choice reading comprehension items without direct writing assessments.
- Who was studied
- N = 1,181 randomized teachers, 3,802 non-randomized school teachers, and 17,153 non-study district teachers across grades 4–8 in 6 urban school districts.
- How
- Cross-year covariance estimation of value-added scores between 2008–09 and 2009–10 to isolate persistent signal variance from temporary noise.
What to do
Calculate cross-year covariance of performance metrics to isolate true signal standard deviation before establishing performance benchmark thresholds.
From the source
"As in the earlier MET project reports, we calculated the 'signal variance' in teachers' true impacts on student achievement by taking the covariance in estimated value-added in 2008-09 and 2009-10. The implied standard deviation in 'true teacher effects' ranged from .16 to .18 in math and from .10 to .12 in ELA."
Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment