Assessment Format Volatility in Literacy Value-Added Estimation
RCTClinical Trial
State English Language Arts (ELA) value-added metrics display substantially higher year-to-year volatility and lower predictive stability than mathematics metrics, reducing their standalone reliability in teacher evaluation systems.
Picture this
Evaluating a runner's stamina using a multiple-choice quiz about running technique instead of measuring actual running distance introduces measurement noise that causes annual test scores to fluctuate unpredictably.
What the evidence says
ELA value-added stability coefficients between consecutive years were only 0.350 in elementary and 0.117 in middle school, compared to 0.396 and 0.512 in math. This instability stemmed from state ELA tests relying primarily on multiple-choice reading comprehension items that fail to directly capture teacher impacts on student writing.
- Who was studied
- 782 elementary (grades 4–5) and 559 middle school (grades 6–8) math and ELA teachers across 6 urban public school districts.
- How
- Cross-year linear regression comparing 2009–10 value-added scores against 2008–09 value-added scores across subjects.
What to do
Apply larger empirical Bayes shrinkage factors and integrate direct open-ended writing assessments when evaluating reading and language arts instructors.
From the source
"In the 2010 MET project report, we speculated that the lower reliability of the state ELA value-added gains was due to the reliance on multiple choice reading comprehension questions in the state ELA tests, which do not capture teacher effects on student writing."
Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment