Predicting counterfactual outcomes aggregates multiple baseline dimensions of disadvantage into a single unbiased index.

Download image · 1200 × 630 PNG

Research analysis

Compare effects by predicted earnings

  1. Train earnings model on control subsample
  2. Split predicted earnings at the median
  3. Estimate group effects and test equality

Illustrative summary of the source-described split-sample analysis: use pre-specified baseline covariates, bootstrap standard errors and an F-test of high- versus low-predicted group effects. No earnings result is shown.

See source →

How

When to use

Evaluators need to test whether program gains are concentrated among disadvantaged populations without cherry-picking subgroups post hoc. Standard subgroup splitting on single baseline variables can produce false positives and fails to aggregate multidimensional disadvantage. Split-sample endogenous stratification groups individuals by predicted baseline performance to evaluate equity impacts systematically.

Stratify trial participants into predicted performance tiers using baseline covariates to evaluate heterogeneous treatment impacts.

  1. Select baseline demographic and socioeconomic covariates specified in the pre-analysis plan to predict long-term outcomes.
  2. Estimate an Ordinary Least Squares linear regression of endline control group earnings on baseline covariates using a training sub-sample.
  3. Predict counterfactual endline outcomes for all study participants based on their baseline characteristics.
  4. Split the full experimental sample into high-predicted and low-predicted outcome strata along the median.
  5. Estimate treatment effects separately across high-predicted and low-predicted strata using a saturated model.
  6. Calculate bootstrap standard errors to account for first-stage outcome prediction uncertainty.
  7. Conduct an F-test comparing treatment effect equality between high-predicted and low-predicted strata.

Why it works

This isolates whether active labor market policies compress inequality by delivering larger relative gains to job-seekers with the lowest baseline earning prospects.

Example

Researchers evaluated 3,052 young job-seekers in Addis Ababa, Ethiopia across 4-year follow-up rounds. The split-sample endogenous stratification showed that the job application workshop increased 4-year earnings by 467.1 Birr among low-predicted earners, reducing the earnings gap between high- and low-predicted groups from 142% down to 54%.

About the study

Participants

N = 3,052 young job-seekers aged 18–29 in Addis Ababa, Ethiopia evaluated over 4 years.

Study Design

Split-sample endogenous stratification following Abadie et al. (2017), predicting 2018 earnings using baseline covariates and bootstrapping standard errors across high and low predicted strata.

Results

Workshop treatment effects were significantly higher for the low-predicted-earnings group (467.1 Birr increase, p < 0.01) than the high-predicted group (-99.0 Birr, p > 0.10; p = 0.0696 for equality), compressing the baseline earnings gap from 142% to 54%.

Source

See source →