Original source
Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment (2013)
research report
Search for this source ↗©2013 Bill & Melinda Gates Foundation. All Rights Reserved.
Permission for School’s use is unestablished.
Full source notice
©2013 Bill & Melinda Gates Foundation. All Rights Reserved.
Used in these cards
Alternative Classroom Observation Rubric Sensitivity and Generalizability
"As described in Kane and Staiger (2012), the project collected teacher scores on a number of other instruments as well: the Classroom Assessment Scoring System (CLASS), the Protocol for Language Arts Teacher Observations (PLATO), the Mathematical Quality of Instruction (MQI), and the UTeach Teacher Observation Protocol (UTOP). However, since one of our goals was to approximate what a given district or state could do with its data, we used only one instrument, Framework for Teaching, in this analysis."
Assessment Format Volatility in Literacy Value-Added Estimation
"In the 2010 MET project report, we speculated that the lower reliability of the state ELA value-added gains was due to the reliance on multiple choice reading comprehension questions in the state ELA tests, which do not capture teacher effects on student writing."
Attrition Independence from Assigned Treatment Effectiveness
"There was no statistically significant relationship between the assigned teacher's measured effectiveness and the availability of the subsequent state achievement measures."
Between-School Generalizability Limit of Within-School RCTs
"Because we could not randomly assign students or teachers to a different school site, our study does not allow us to investigate the validity of the measures of effectiveness for gauging differences across schools."
Classroom Observation Standalone Predictive Validity
"The results are qualitatively similar to those we saw with the full composite: the estimates of $\gamma_{1}$ are large and not significantly different from one. However, because classroom observations and student perception surveys are much less strongly predictive of student achievement gains, the standard errors on these are considerably higher (roughly of.3 rather than.149 for the value-added measure)."
Demographic Covariate Inclusion in Value-Added Modeling
"Although the coefficients on $\hat{\tau}_{j}^{0}$ and $\hat{\tau}_{j}^{1}-\hat{\tau}_{j}^{0}$ are both statistically different from zero and not statistically different from one, the component associated with student demographics, $\hat{\tau}_{j}^{2}-\hat{\tau}_{j}^{1}$, is not statistically different from zero."
District-Level Structural Heterogeneity in Experimental Compliance
"Due to higher levels of compliance with the randomly assigned rosters, the effectiveness of the assigned teachers were more likely to filter through to actual teacher effectiveness in Dallas and Charlotte-Mecklenburg (.744 and.517, respectively) than in Hillsborough, Memphis, and New York (.380,.277, and.157, respectively)."
Empirical Bayes Shrinkage Estimation for Volatility Adjustment
"When there is a lot of error, we should be less willing to 'go out on a limb' with large positive or large negative predictions. We should rein in the estimates, shrinking both the large positive impacts and the large negative impacts back toward zero."
High School End-of-Course Teacher Effectiveness Validity
"In the first column, we combine the results across three subjects: math, ELA, and biology. The instrumental variables estimate of the effect of actual teacher effectiveness was.828, with a standard error of.29."
Instrumental Variables Estimation for Experimental Non-Compliance
"In the end, many students ended up with an 'actual' teacher who was different from their 'assigned teacher.' Nevertheless, we were able to exploit the random assignment to infer the effect of the 'actual' teacher, using a two-step process known as 'instrumental variables' or 'IV estimation.'"
Intent-to-Treat (ITT) Reduced-Form Effect Estimation
"In instrumental variables estimation, this is commonly referred to as the 'reduced form' or 'intent-to-treat' model."
Joint Hypothesis Balance Testing in Pooled Randomization Blocks
Seemingly Unrelated Regressions (SURE) for Joint Covariate Balance
"The joint hypothesis test was estimated using the 'seemingly unrelated regressions' or SURE model proposed by Zellner (1962)."
Multi-Measure Composite Teacher Evaluation
"First, we used the data collected during 2009-10 to build a composite measure of teaching effectiveness, combining all three measures to predict a teacher's impact on another group of students."
Teacher Achievement Impact on Student Classroom Enjoyment
"His or her students were also.84 standard deviations more likely to self-report that they enjoyed being in the class."
Post-Randomization Peer Compositional Stability
"In sum, despite the movement into and out of the randomized classrooms, the peers to whom randomized students were exposed were not systematically different based on the effectiveness of their randomly assigned teacher."
Peer Effects Overcontrolling in Value-Added Models
"In other words, the component removed by controls for peer effects does seem to reflect causal teacher effects-not just factors outside a teacher's control."
Persistent Student Baseline Sorting Covariance
"We measure persistent sorting using the covariance in a teacher's students' baseline test scores over time. If a teacher gets a group of high-scoring students one year but not the next, that may or may not have occurred by chance that year, but it is not persistent sorting."
Within-School Exchangeable Rosters and Randomization Blocks
"We call these groups of teachers 'randomization blocks.' Schools then constructed rosters of students for the randomization blocks and submitted them to the study team. Within each randomization block, analysts at the RAND Corporation randomly assigned each of the rosters to the participating MET project teachers."
Signal Variance Decomposition of Teacher Effects
"As in the earlier MET project reports, we calculated the 'signal variance' in teachers' true impacts on student achievement by taking the covariance in estimated value-added in 2008-09 and 2009-10. The implied standard deviation in 'true teacher effects' ranged from.16 to.18 in math and from.10 to.12 in ELA."
Student Sorting Bias and Baseline Score Adjustment
Unadjusted Status Score Bias in Performance Evaluation
"In other words, if we used end-of-year scores to assess teachers' effectiveness and failed to adjust for students' prior achievement, then we would be overstating the differences between teachers."
Student Perception Survey Standalone Predictive Validity
"However, because classroom observations and student perception surveys are much less strongly predictive of student achievement gains, the standard errors on these are considerably higher (roughly of.3 rather than.149 for the value-added measure)."
Supplemental Conceptual Assessment Transfer
"When a teacher was predicted to improve student achievement on state tests by one standard deviation, his or her students' performance on the supplemental assessments increased by.7 standard deviations on average."
Value-Added Extreme Outlier Trimming and Anomaly Detection
"Accordingly, for the remainder of the paper, we dropped teachers in the top 1 percent of the value-added distribution in math or ELA."
Rank-Based Van Der Waerden Score Standardization
"We standardized test scores using a rank-based standardization method or van der Waerden scores (Conover, 1999), which first ranked students based on the original test score and then assigned a standardized score based on the average score for students with that rank if the underlying scores were standard normal".
Random Effects and Fixed Effects Value-Added Equivalence
"In unpublished work, we have found such a specification yielded value-added estimates that are highly correlated to those from a teacher fixed effects specification, since the lion's share of the variation in student characteristics is within classroom as opposed to between classrooms (Ehlert et al., under review, report similar results with data in Missouri)."