Construct Sensitivity Divergence in Non-Cognitive Evaluation
RCTClinical Trial
Evaluating targeted non-cognitive interventions with broad psychological inventories can mask true skill improvements due to instrument misalignment with specific behavioral domains.
Picture this
Evaluating a focused emotional self-control workshop with a broad psychological test is like using a general physical check-up to measure the impact of a targeted leg-strengthening program; general health metrics show minimal change, but direct leg-strength tests reveal clear progress.
What the evidence says
Intention-to-treat estimates showed a statistically significant improvement of 0.165 to 0.175 standard deviations on targeted PLANEA scores (p < 0.05), compared to a statistically insignificant 0.017 to 0.027 standard deviation effect on the broad BarOn inventory.
- Who was studied
- N = 1,241 to 1,422 10th-grade public high school students in Mexico City evaluated at the end of the academic year.
- How
- Exploratory Factor Analysis deriving standardized Bartlett scores to compare targeted PLANEA sub-scales against general BarOn Emotional Quotient Inventory scales within an RCT.
What to do
Select psychometric evaluation instruments that measure the explicit sub-skills targeted by an intervention rather than relying on broad emotional intelligence inventories.
From the source
"The effect on the PLANEA score is large and statistically significant, about 0.17 and 0.18 standard deviations (sd)... When we look at the impact on the BarOn inventory, we find an effect that is small for both treatment groups and statistically not significant."
Addressing High School Dropouts with a Scalable Intervention: The Case of PODER