Subjecting treatment effect estimates to extreme outcome imputations verifies whether long-term findings survive worst-case differential attrition assumptions.

Download image · 1200 × 630 PNG

Research analysis

Test sensitivity to missing outcomes

  1. Predict outcomes from control baseline data
  2. Impute missing outcomes under penalties
  3. Re-estimate effects across scenarios

Illustrative summary of the source-described attrition analysis: test 0.25 and 0.5 control-standard-deviation penalties, favouring missing control outcomes and penalising missing treatment outcomes; also examine gender and education subgroups.

See source →

How

When to use

Longitudinal field experiments often experience differential participant dropouts across treatment and control groups over multi-year follow-up periods. Differential attrition threatens internal validity by potentially biasing long-run treatment effect estimates. Impute extreme and plausible outcome values for missing respondents to establish robust lower-bound treatment effect estimates.

Test the sensitivity of treatment estimates to differential attrition by imputing missing survey values under increasingly conservative distribution scenarios.

  1. Predict endline outcomes for non-attrited control group respondents using Ordinary Least Squares regression on baseline covariates.
  2. Impute counterfactual missing endline values for all attrited respondents using their predicted baseline values.
  3. Construct asymmetric differential attrition scenarios by adding standard deviation penalties to control missing values while subtracting them from treatment missing values.
  4. Apply conservative group-level mean imputations penalizing missing treatment outcomes by 0.25 and 0.5 control standard deviations.
  5. Tighten outcome bounds by calculating missing value imputations separately across specific gender and education sub-categories.
  6. Re-estimate main intent-to-treat regression models under each missing data scenario to confirm treatment coefficient persistence.

Why it works

Showing that positive treatment impacts persist under penalizing imputations confirms that results are not driven by selective participant dropout.

Example

Researchers tested 4-year earnings gains for 3,052 young job-seekers in Addis Ababa, Ethiopia under multiple attrition scenarios. Imputing predicted earnings with 0.25 and 0.5 standard deviation penalties maintained statistically significant treatment effects of 223.4 Birr and 198.7 Birr.

About the study

Participants

N = 3,052 young job-seekers aged 18–29 in Addis Ababa, Ethiopia evaluated at 4-year endline (15% attrition rate).

Study Design

Sensitivity analysis imputing missing 2018 earnings under predicted, +/-0.25 SD, and +/-0.5 SD control distribution scenarios following Karlan and Valdivia (2011).

Results

The workshop earnings effect remained statistically significant under predicted earnings (248.1 Birr, p < 0.05), +/-0.25 SD predicted earnings (223.4 Birr, p < 0.05), and +/-0.5 SD predicted earnings (198.7 Birr, p < 0.10).

Source

See source →