Research procedure
Adjust for tests of related outcomes
- Group outcomes; estimate p-values
- Calculate correlations and adjust
- Compare adjusted significance levels
Illustrative summary of the source-described Sankoh-McKenzie adjustment across five technology-adoption outcomes. Calculate each outcome’s exponent from outcome count and average correlation excluding that outcome, then transform its p-value. Compare the adjusted values with significance thresholds.
See source →How
When to use
Evaluating agricultural development programs across multiple technology outcomes increases the likelihood of false positive statistical discoveries. Standard single-hypothesis testing risks declaring non-significant inputs as effective interventions. Applying a correlation-adjusted p-value formula accounts for outcome dependencies while controlling overall family-wise error rates.
Adjust estimated p-values across a family of correlated outcome variables to control family-wise error rates during impact evaluations.
- Group individual technology adoption measures into a single family of outcomes.
- Estimate unadjusted p-values for treatment coefficients across each individual outcome in the family using regression models.
- Calculate the average correlation coefficient among all outcomes in the family excluding the specific target outcome.
- Calculate an adjusted outcome exponent using the total outcome count and the calculated average correlation.
- Transform each unadjusted p-value into an adjusted p-value using the outcome exponent formula.
- Compare the adjusted p-values against standard significance thresholds to identify statistically robust treatment effects.
Why it works
Accounting for mean inter-outcome correlation prevents over-correction while guarding against false discoveries across multiple endline tests.
Example
In evaluating the GIZ-ISFM+ project across 2,382 Ethiopian smallholders, researchers applied the Sankoh-McKenzie adjustment across five individual soil fertility practices. Following adjustment, estimated treatment effects for compost, line seeding, and lime remained statistically significant (p < 0.01), whereas p-values for blended fertilizer (p = 0.749 in T1) and improved seeds (p = 0.291 in T2) rose above the 10 percent significance threshold.
About the study
Participants
N = 2,382 smallholder households across 201 microwatersheds in Amhara, Oromia, and Tigray in Ethiopia.
Study Design
Cluster Randomized Controlled Trial with p-values adjusted for multiple hypothesis testing using the Sankoh-McKenzie formulation across 5 technology adoption outcomes.
Results
Following Sankoh-McKenzie adjustment, treatment effects for extension alone (T1) on compost adoption remained at p < 0.001 and line seeding at p = 0.007, while blended fertilizer increased to p = 0.749 and improved seeds to p = 0.092.