Averaging standardized z-scores across related indicators prevents false positive findings caused by testing multiple outcomes independently.

Download image · 1200 × 630 PNG

Research steps

Combine related outcomes into summary indices

  1. Group outcomes; standardise against controls
  2. Reverse negative measures; average z-scores
  3. Estimate treatment effects on each index

Illustrative summary of the source-described unweighted composite indices. Standardisation subtracts the control mean and divides by its standard deviation. The reported human-capital estimate was +0.019 standard deviations, statistically indistinguishable from zero; this does not prove an exact zero.

See source →

How

When to use

Evaluating social programs across dozens of survey and administrative variables creates multiple hypothesis testing problems and risks cherry-picking significant results. Program evaluators struggle to synthesize overall impact across diverse but related behavioral indicators. This method constructs standardized composite z-score summary indices across families of outcomes to evaluate net treatment impacts without multiple testing bias.

Construct standardized composite z-score indices across families of related behavior outcomes following the method of Kling, Liebman, and Katz (2007).

  1. Evaluators group related individual outcome variables into distinct thematic categories, including human capital investment, job search intensity, and employment quality.
  2. Staff standardize each individual outcome variable by subtracting the control group mean and dividing by the control group standard deviation.
  3. Evaluators reverse the sign of negative outcome variables so that higher values consistently represent desirable outcomes across all index components.
  4. Staff average the standardized component z-scores for each participant to create a single composite summary index value.
  5. Evaluators run treatment effect regressions on the composite summary indices to assess aggregate program impact across outcome families.

Why it works

This provides a single, statistically robust measure of aggregate program impact across complex behavioral domains.

Example

In France, evaluators combined measures of vocational training, apprenticeships, internships, and driver's license courses into a single human capital investment index across 5,498 youth, finding that cash transfers yielded a treatment effect of +0.019 standard deviations, which was statistically indistinguishable from zero.

About the study

Participants

5,498 young unskilled jobseekers aged 18–22 across 82 Job Youth Centers in France (2,661 assigned to cash transfers and 2,837 to control).

Study Design

Randomized controlled trial aggregating administrative and survey variables into standardized summary indices normalized against control group distributions.

Results

The human capital investment index showed an ITT treatment effect of +0.019 standard deviations (p > 0.10) and the job search effort index showed an ITT effect of -0.010 standard deviations (p > 0.10), proving that cash transfers had a precisely estimated zero impact on overall employability efforts.

Source

See source →