Research steps
Adjust for multiple tests
- Group outcomes; estimate p-values
- Sort and rank within each family
- Apply rank × q / M threshold
Illustrative summary of the source-described Benjamini–Hochberg procedure. Use the largest qualifying rank; reject through that rank and report adjusted q-values. No results shown.
See source →How
When to use
Evaluating workplace interventions across multiple survey outcomes and performance metrics risks false positive findings due to multiple hypothesis testing. Running dozens of individual hypothesis tests increases the probability of erroneously rejecting true null hypotheses by chance alone. Apply false discovery rate control using the Benjamini-Hochberg procedure to adjust p-values and confirm statistical significance across family groups of outcomes.
Adjust p-values across family sets of experimental hypotheses to bound the expected proportion of false positives at level q using the Benjamini-Hochberg procedure.
- Group related trial outcomes into coherent hypothesis families, such as productivity metrics, workplace survey outcomes, financial behaviors, and personality traits.
- Estimate individual treatment regressions for each outcome variable within a hypothesis family to obtain raw p-values.
- Sort the M individual p-values within each family in ascending numerical order.
- Assign an integer rank index from one to M to each sorted p-value.
- Calculate the false discovery rate critical threshold for each rank by multiplying the rank by target q and dividing by total hypotheses M.
- Identify the largest rank where the observed p-value is less than or equal to the calculated critical threshold.
- Reject all hypotheses up to that threshold rank and report adjusted false discovery rate q-values for all family outcomes.
Why it works
Grouping outcomes into domain-specific families ensures that multiple testing adjustments evaluate related behavioral constructs together.
Example
Researchers testing soft skills training effects across 27 survey and workplace outcomes grouped hypotheses into six thematic families, including productivity, financial behavior, and personality traits. Applying Benjamini-Hochberg false discovery rate adjustments confirmed that post-program productivity efficiency gains remained statistically significant with a q-value of 0.049 and skill-peer self-assessments remained significant with a q-value of 0.006.
About the study
Participants
2,703 female garment workers across 112 production lines in 5 export factories in Bengaluru, India (621 survey respondents).
Study Design
Two-stage randomized controlled trial using Anderson's implementation of the Benjamini-Hochberg false discovery rate procedure across six outcome families (productivity, workplace survey, financial behavior, entitlements, personality, and mental health).
Results
Multiple testing corrections confirmed post-treatment efficiency gains remained statistically significant (q-value = 0.049; point estimate 0.108), task complexity SAM increases remained significant (q-value = 0.049; point estimate 0.0384), and skill-peer self-assessment remained significant (q-value = 0.006; point estimate 0.158), while extraversion became marginally insignificant (q-value = 0.108).