Standard Bonferroni corrections assume independent outcomes and become overly conservative when outcomes are correlated.

Download image · 1200 × 630 PNG

Research procedure

Adjust for tests of related outcomes

  1. Group outcomes; estimate p-values
  2. Calculate correlations and adjust
  3. Compare adjusted significance levels

Illustrative summary of the source-described Sankoh-McKenzie adjustment across five technology-adoption outcomes. Calculate each outcome’s exponent from outcome count and average correlation excluding that outcome, then transform its p-value. Compare the adjusted values with significance thresholds.

See source →

How

When to use

Evaluating agricultural development programs across multiple technology outcomes increases the likelihood of false positive statistical discoveries. Standard single-hypothesis testing risks declaring non-significant inputs as effective interventions. Applying a correlation-adjusted p-value formula accounts for outcome dependencies while controlling overall family-wise error rates.

Adjust estimated p-values across a family of correlated outcome variables to control family-wise error rates during impact evaluations.

  1. Group individual technology adoption measures into a single family of outcomes.
  2. Estimate unadjusted p-values for treatment coefficients across each individual outcome in the family using regression models.
  3. Calculate the average correlation coefficient among all outcomes in the family excluding the specific target outcome.
  4. Calculate an adjusted outcome exponent using the total outcome count and the calculated average correlation.
  5. Transform each unadjusted p-value into an adjusted p-value using the outcome exponent formula.
  6. Compare the adjusted p-values against standard significance thresholds to identify statistically robust treatment effects.

Why it works

Accounting for mean inter-outcome correlation prevents over-correction while guarding against false discoveries across multiple endline tests.

Example

In evaluating the GIZ-ISFM+ project across 2,382 Ethiopian smallholders, researchers applied the Sankoh-McKenzie adjustment across five individual soil fertility practices. Following adjustment, estimated treatment effects for compost, line seeding, and lime remained statistically significant (p < 0.01), whereas p-values for blended fertilizer (p = 0.749 in T1) and improved seeds (p = 0.291 in T2) rose above the 10 percent significance threshold.

About the study

Participants

N = 2,382 smallholder households across 201 microwatersheds in Amhara, Oromia, and Tigray in Ethiopia.

Study Design

Cluster Randomized Controlled Trial with p-values adjusted for multiple hypothesis testing using the Sankoh-McKenzie formulation across 5 technology adoption outcomes.

Results

Following Sankoh-McKenzie adjustment, treatment effects for extension alone (T1) on compost adoption remained at p < 0.001 and line seeding at p = 0.007, while blended fertilizer increased to p = 0.749 and improved seeds to p = 0.092.

Source

See source →