Multiple Testing Familywise Correction
RCTReview
Testing heterogeneous treatment effects across multiple subgroup dimensions (e.g., cognitive ability, prior experience, research background) increases the risk of false positive statistical discoveries (Type I error), requiring formal adjustments to confirm valid subgroup interactions.
Picture this
Imagine rolling a pair of dice repeatedly to get a double six. If you roll them enough times across different games, you will eventually hit a double six by pure chance. Raising the threshold for what counts as a true win based on how many total rolls were made prevents mistaking random luck for genuine skill.
What the evidence says
The interaction between treatment and prior international employer experience remains statistically significant after Bonferroni correlation adjustment (adjusted p = 0.041), whereas ability interaction significance is slightly attenuated (adjusted p = 0.069 to 0.082), confirming international experience as a robust subgroup moderator.
- Who was studied
- N = 227 urban male youth job seekers in Lilongwe, Malawi.
- How
- Econometric sensitivity testing applying standard Bonferroni adjustments and Sankoh-Huque-Dubey correlation-corrected Bonferroni adjustments to interaction p-values across baseline ability and experience subgroups.
What to do
Apply correlation-adjusted Bonferroni corrections to interaction p-values when evaluating multiple dimensions of treatment heterogeneity in experimental data.
From the source
"We present the Bonferroni-adjusted p-values as well as Bonferroni adjustments that correct for correlation (Sankoh, Huque, and Dubey 1997)."
b23e842f-67cd-47d7-9b15-b6b6c8e557a2-Employment Exposure- Employment and Wage Effects in Urban Malawi.pdf