Original source
Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment (2013)
research report
Search for this source ↗©2013 Bill & Melinda Gates Foundation. All Rights Reserved.
Permission for School’s use is unestablished.
Full source notice
©2013 Bill & Melinda Gates Foundation. All Rights Reserved.
Used in these cards
Evaluate subject-specific observational rubrics (such as MQI and PLATO) alongside general frameworks (such as FFT) when designing diagnostic feedback and evaluation systems.
"As described in Kane and Staiger (2012), the project collected teacher scores on a number of other instruments as well: the Classroom Assessment Scoring System (CLASS), the Protocol for Language Arts Teacher Observations (PLATO), the Mathematical Quality of Instruction (MQI), and the UTeach Teacher Observation Protocol (UTOP). However, since one of our goals was to approximate what a given district or state could do with its data, we used only one instrument, Framework for Teaching, in this analysis."
Apply larger empirical Bayes shrinkage factors and integrate direct open-ended writing assessments when evaluating reading and language arts instructors.
"In the 2010 MET project report, we speculated that the lower reliability of the state ELA value-added gains was due to the reliance on multiple choice reading comprehension questions in the state ELA tests, which do not capture teacher effects on student writing."
Regress outcome data availability indicators on assigned treatment ratings to test for attrition bias in multi-site randomized field trials.
"There was no statistically significant relationship between the assigned teacher's measured effectiveness and the availability of the subsequent state achievement measures."
Limit policy claims regarding teacher value-added validity to within-school comparisons unless inter-school random reassignment data become available.
"Because we could not randomly assign students or teachers to a different school site, our study does not allow us to investigate the validity of the measures of effectiveness for gauging differences across schools."
Incorporate structured classroom observation rubrics into multi-measure composite evaluation systems alongside test-based value-added metrics to maintain qualitative feedback while reducing overall predictive variance.
"The results are qualitatively similar to those we saw with the full composite: the estimates of $\gamma_{1}$ are large and not significantly different from one. However, because classroom observations and student perception surveys are much less strongly predictive of student achievement gains, the standard errors on these are considerably higher (roughly of.3 rather than.149 for the value-added measure)."
Prioritize student baseline achievement controls as the primary bias-reduction mechanism while carefully evaluating district policy trade-offs before adding or prohibiting student demographic covariates.
"Although the coefficients on $\hat{\tau}_{j}^{0}$ and $\hat{\tau}_{j}^{1}-\hat{\tau}_{j}^{0}$ are both statistically different from zero and not statistically different from one, the component associated with student demographics, $\hat{\tau}_{j}^{2}-\hat{\tau}_{j}^{1}$, is not statistically different from zero."
Audit local institutional constraints and allow first-stage compliance rates to vary by site when evaluating multi-district field experiments.
"Due to higher levels of compliance with the randomly assigned rosters, the effectiveness of the assigned teachers were more likely to filter through to actual teacher effectiveness in Dallas and Charlotte-Mecklenburg (.744 and.517, respectively) than in Hillsborough, Memphis, and New York (.380,.277, and.157, respectively)."
Multiply single-year performance metrics by cross-year stability coefficients before deploying estimates for predictive modeling or high-stakes decision-making.
"When there is a lot of error, we should be less willing to 'go out on a limb' with large positive or large negative predictions. We should rein in the estimates, shrinking both the large positive impacts and the large negative impacts back toward zero."
Deploy standardized end-of-course subject assessments (such as ACT QualityCore) to evaluate and validate high school teacher effectiveness across departmentalized subject areas.
"In the first column, we combine the results across three subjects: math, ELA, and biology. The instrumental variables estimate of the effect of actual teacher effectiveness was.828, with a standard error of.29."
Apply initial randomized assignment rosters as an instrumental variable in a two-stage LIML model to isolate unbiased causal treatment effects when experimental compliance is incomplete.
"In the end, many students ended up with an 'actual' teacher who was different from their 'assigned teacher.' Nevertheless, we were able to exploit the random assignment to infer the effect of the 'actual' teacher, using a two-step process known as 'instrumental variables' or 'IV estimation.'"
Calculate the intent-to-treat reduced-form regression of outcomes on assigned treatment variables to establish the baseline causal impact of program assignment prior to instrumental variable adjustments.
"In instrumental variables estimation, this is commonly referred to as the 'reduced form' or 'intent-to-treat' model."
Execute a SURE joint hypothesis test across all baseline covariates simultaneously to verify experimental randomization balance in blocked multi-site trials.
Specify Zellner's SURE system estimation to execute joint hypothesis testing across all baseline covariates when validating randomization balance in multi-site RCTs.
"The joint hypothesis test was estimated using the 'seemingly unrelated regressions' or SURE model proposed by Zellner (1962)."
Construct a weighted evaluation composite combining prior achievement value-added, structured classroom observation scores, and student perception surveys to forecast instructor effectiveness.
"First, we used the data collected during 2009-10 to build a composite measure of teaching effectiveness, combining all three measures to predict a teacher's impact on another group of students."
Include student self-reported classroom enjoyment in multi-dimensional evaluation systems to measure instructor impacts on positive classroom climate alongside academic growth.
"His or her students were also.84 standard deviations more likely to self-report that they enjoyed being in the class."
Assess the baseline characteristics of actual end-of-year classroom peers and regress them on assigned treatment intensity to rule out peer compositional contamination.
"In sum, despite the movement into and out of the randomized classrooms, the peers to whom randomized students were exposed were not systematically different based on the effectiveness of their randomly assigned teacher."
Omit classroom-level baseline mean achievement controls (peer effect covariates) from within-school teacher value-added specifications to prevent subtracting legitimate teacher impacts.
"In other words, the component removed by controls for peer effects does seem to reflect causal teacher effects-not just factors outside a teacher's control."
Calculate multi-year baseline score covariance across instructor rosters to measure persistent student sorting before implementing unadjusted evaluation metrics.
"We measure persistent sorting using the covariance in a teacher's students' baseline test scores over time. If a teacher gets a group of high-scoring students one year but not the next, that may or may not have occurred by chance that year, but it is not persistent sorting."
Restrict random assignment blocks in educational field experiments to subset groups of instructors sharing identical grade, subject certification, and schedule period availability.
"We call these groups of teachers 'randomization blocks.' Schools then constructed rosters of students for the randomization blocks and submitted them to the study team. Within each randomization block, analysts at the RAND Corporation randomly assigned each of the rosters to the participating MET project teachers."
Calculate cross-year covariance of performance metrics to isolate true signal standard deviation before establishing performance benchmark thresholds.
"As in the earlier MET project reports, we calculated the 'signal variance' in teachers' true impacts on student achievement by taking the covariance in estimated value-added in 2008-09 and 2009-10. The implied standard deviation in 'true teacher effects' ranged from.16 to.18 in math and from.10 to.12 in ELA."
Control for prior baseline student achievement test scores in evaluation models to eliminate selection bias caused by classroom sorting.
Replace raw unadjusted status test scores with prior achievement-adjusted growth models when constructing instructor evaluation frameworks.
"In other words, if we used end-of-year scores to assess teachers' effectiveness and failed to adjust for students' prior achievement, then we would be overstating the differences between teachers."
Incorporate student perception surveys as a supporting component within a multi-measure evaluation composite rather than relying on surveys as a sole decision-making metric.
"However, because classroom observations and student perception surveys are much less strongly predictive of student achievement gains, the standard errors on these are considerably higher (roughly of.3 rather than.149 for the value-added measure)."
Administer conceptual and open-ended supplemental assessments to verify that value-added achievement gains reflect transferable higher-order problem-solving skills rather than narrow test-preparation effects.
"When a teacher was predicted to improve student achievement on state tests by one standard deviation, his or her students' performance on the supplemental assessments increased by.7 standard deviations on average."
Implement a 1% extreme value trimming protocol on estimated teacher value-added scores before deploying them in evaluation or predictive validity research models.
"Accordingly, for the remainder of the paper, we dropped teachers in the top 1 percent of the value-added distribution in math or ELA."
Apply rank-based van der Waerden score normalization to raw test scores before pooling student achievement metrics across disparate assessment systems.
"We standardized test scores using a rank-based standardization method or van der Waerden scores (Conover, 1999), which first ranked students based on the original test score and then assigned a standardized score based on the average score for students with that rank if the underlying scores were standard normal".
Deploy two-step residual-averaging random-effects value-added models with classroom-mean controls as a computationally efficient and statistically equivalent alternative to teacher fixed-effects models.
"In unpublished work, we have found such a specification yielded value-added estimates that are highly correlated to those from a teacher fixed effects specification, since the lion's share of the variation in student characteristics is within classroom as opposed to between classrooms (Ehlert et al., under review, report similar results with data in Missouri)."