Pooling multiple related outcomes into a single weighted index prevents overtesting and false discoveries across individual survey questions.

Download image · 1200 × 630 PNG

Research procedure

Combine related survey outcomes

  1. Align direction; centre and standardise outcomes
  2. Calculate control-group covariance and weights
  3. Combine measures into a weighted index

Illustrative summary of the source procedure. Centre on control means and divide by control standard deviations before inverse-covariance weighting. The source combines six gender items and four poverty items into their respective indices.

See source →

How

When to use

Researchers need to evaluate the impact of an intervention on complex political attitudes measured across multiple survey questions. Running separate statistical tests on numerous individual survey items increases the risk of false positives from multiple testing. Constructing a single weighted summary index combines multiple related survey items into an overall measure while increasing statistical power.

Combine multiple related survey questions into a single standardized index using inverse covariance weighting.

  1. Orient every individual survey outcome so that the positive direction indicates higher agreement or support for the policy issue.
  2. Demean all individual survey outcomes by subtracting the control group mean from each respondent's value.
  3. Standardize all outcomes by dividing by the control group standard deviation to place all measures on a comparable scale.
  4. Compute the covariance matrix of the standardized outcomes within the control group.
  5. Calculate inverse-covariance weights to assign optimal linear combination weights that capture the common underlying factor.
  6. Calculate the weighted average across all component measures for each respondent to construct the final summary index.

Why it works

Weighting by the inverse covariance matrix extracts a common latent factor while providing greater statistical power than separate single-variable tests.

Example

Researchers evaluated the impact of Akbayan and Umalab Ka town-hall meetings on voter attitudes across six gender discrimination questions and four poverty policy questions in post-election surveys in the Philippines. Combining individual survey questions into inverse-covariance-weighted indices showed that exposure to Akbayan meetings increased gender awareness by 0.425 standard deviation units overall. In contrast, exposure to Umalab Ka meetings increased poverty concern by 0.132 standard deviation units overall.

About the study

Participants

1,082 surveyed citizens across treatment and control barangays in Metro Manila and Calabarzon, Philippines, responding to 6 gender attitude items and 4 poverty attitude items.

Study Design

Standardized summary index construction using inverse-covariance-weighted linear combinations to evaluate treatment effects on composite attitudinal outcomes.

Results

Inverse-covariance summary indexing revealed that deliberative town halls increased gender awareness by 0.425 standard deviation units for Akbayan and increased poverty concern by 0.132 standard deviation units for Umalab Ka.

Source

See source →