In Simple Words
Evaluating a program using only average results is like measuring the average temperature of a room with one frozen wall and one burning wall and concluding the room is comfortable. Machine learning acts like a thermal camera, splitting the room into sections to reveal the extreme cold and extreme heat hiding behind the average.
Cairo job seekers grouped by predicted response
- Highest predicted response
- Application effect: +5.4 percentage points (uncertain)
- Lowest predicted response
- Application effect: −15.5 percentage points
For Experiment 1 social-stigma messaging, the source reports treatment effects of +5.4 percentage points in the highest predicted-response group (not statistically significant) and −15.5 points in the lowest. The direct top-versus-bottom test is significant at p = 0.009. These are effects within model-defined groups, not absolute application rates.
See source →Research question
Standard average treatment effect (ATE) estimates frequently fail to detect meaningful behavioral responses because opposing positive and negative reactions across sub-populations cancel each other out in aggregate data.
Participants
N = 1,460 to 1,470 (Experiment 1) and N = 768 (Experiment 2) unemployed/underemployed youth aged 18–35 in Cairo, Egypt.
Study Design
Chernozhukov et al. (2022) generic machine learning framework employing 100 iterations of split-sample cross-validation across Elastic Net, Boosted Trees, Random Forest, and Neural Networks to calculate Sorted Group Average Treatment Effects (GATES).
Results
While overall average treatment effects were small and statistically insignificant (-1.4 to -2.7 percentage points, p > 0.10), generic ML inference uncovered significant treatment effect gaps between the top and bottom quintiles of 20.9 percentage points for social stigma (p = 0.009) and 12.6 percentage points for salient stigma outreach (p = 0.022).
Takeaway
Apply agnostic machine learning split-sample validation to estimate Sorted Group Average Treatment Effects before concluding that an economic or social intervention lacks impact.
Source and citation
See source →Read alongside the research
Plain-English glossary
Original explanations, informed by the references below. Examples are illustrations, not data from the studies above. The research text stays unchanged.
- Percentage points
The difference between two percentages.
ExampleA rise from 40% to 50% is 10 percentage points. Relative to the starting value, that is a 25% increase.
Office for National Statistics · Percentages ↗Read more about percentage points →