Value-Added Extreme Outlier Trimming and Anomaly Detection
RCTClinical Trial
Statistical value-added models are vulnerable to extreme outliers caused by testing anomalies, administrative errors, or cheating, which distort predictive validity models.
Picture this
If a speedometer occasionally flashes 300 mph due to a sensor glitch, including that reading in an auto safety study will ruin the entire analysis. Trimming top and bottom extreme spikes ensures the evaluation reflects true vehicle performance rather than sensor errors.
What the evidence says
In English language arts, including an extreme outlier teacher with a 1.666 SD value-added swing dropped the predictive coefficient to an imprecise 0.375 (SE = 0.264); trimming the top 1% restored the ELA coefficient to 0.697 (p < 0.01, SE = 0.213) and overall composite predictive coefficient to 0.955 (SE = 0.123).
- Who was studied
- N = 27,790 total randomized student observations reduced to N = 27,255 after 1% trimming across 6 urban districts.
- How
- Sensitivity analysis testing LIML IV predictive estimates across full sample versus samples trimmed at top 0.5%, 1%, 2%, and 3% of teacher value-added distribution, alongside multi-year classroom score tracking.
What to do
Implement a 1% extreme value trimming protocol on estimated teacher value-added scores before deploying them in evaluation or predictive validity research models.
From the source
"Accordingly, for the remainder of the paper, we dropped teachers in the top 1 percent of the value-added distribution in math or ELA."
Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment