Unadjusted Status Score Bias in Performance Evaluation
RCTClinical Trial
Evaluators who rely on unadjusted end-of-year student achievement scores to evaluate instructors fail to separate baseline student skill levels from actual teaching quality, creating severe predictive bias.
Picture this
Evaluating a marathon coach based purely on how fast their runners cross the finish line—without knowing if those runners started as elite athletes or complete beginners—evaluates runner selection rather than coaching quality.
What the evidence says
Unadjusted end-of-year score ratings yielded a predictive coefficient of 0.228 (p < 0.01, SE = 0.042), significantly overstating true teacher effectiveness differences compared to baseline-adjusted growth models (coefficient = 0.955).
- Who was studied
- N = 27,255 randomized students in grades 4–8 across 619 randomization blocks in 6 urban school districts.
- How
- LIML IV estimation comparing unadjusted end-of-year student score ratings from prior years against post-randomization student test achievement.
What to do
Replace raw unadjusted status test scores with prior achievement-adjusted growth models when constructing instructor evaluation frameworks.
From the source
"In other words, if we used end-of-year scores to assess teachers' effectiveness and failed to adjust for students' prior achievement, then we would be overstating the differences between teachers."
Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment