Research analysis
Test sensitivity to missing outcomes
- Predict outcomes from control baseline data
- Impute missing outcomes under penalties
- Re-estimate effects across scenarios
Illustrative summary of the source-described attrition analysis: test 0.25 and 0.5 control-standard-deviation penalties, favouring missing control outcomes and penalising missing treatment outcomes; also examine gender and education subgroups.
See source →How
When to use
Longitudinal field experiments often experience differential participant dropouts across treatment and control groups over multi-year follow-up periods. Differential attrition threatens internal validity by potentially biasing long-run treatment effect estimates. Impute extreme and plausible outcome values for missing respondents to establish robust lower-bound treatment effect estimates.
Test the sensitivity of treatment estimates to differential attrition by imputing missing survey values under increasingly conservative distribution scenarios.
- Predict endline outcomes for non-attrited control group respondents using Ordinary Least Squares regression on baseline covariates.
- Impute counterfactual missing endline values for all attrited respondents using their predicted baseline values.
- Construct asymmetric differential attrition scenarios by adding standard deviation penalties to control missing values while subtracting them from treatment missing values.
- Apply conservative group-level mean imputations penalizing missing treatment outcomes by 0.25 and 0.5 control standard deviations.
- Tighten outcome bounds by calculating missing value imputations separately across specific gender and education sub-categories.
- Re-estimate main intent-to-treat regression models under each missing data scenario to confirm treatment coefficient persistence.
Why it works
Showing that positive treatment impacts persist under penalizing imputations confirms that results are not driven by selective participant dropout.
Example
Researchers tested 4-year earnings gains for 3,052 young job-seekers in Addis Ababa, Ethiopia under multiple attrition scenarios. Imputing predicted earnings with 0.25 and 0.5 standard deviation penalties maintained statistically significant treatment effects of 223.4 Birr and 198.7 Birr.
About the study
Participants
N = 3,052 young job-seekers aged 18–29 in Addis Ababa, Ethiopia evaluated at 4-year endline (15% attrition rate).
Study Design
Sensitivity analysis imputing missing 2018 earnings under predicted, +/-0.25 SD, and +/-0.5 SD control distribution scenarios following Karlan and Valdivia (2011).
Results
The workshop earnings effect remained statistically significant under predicted earnings (248.1 Birr, p < 0.05), +/-0.25 SD predicted earnings (223.4 Birr, p < 0.05), and +/-0.5 SD predicted earnings (198.7 Birr, p < 0.10).