Sorting p-values and adjusting thresholds controls the expected proportion of false positives among all rejected null hypotheses without overly penalizing statistical power.

Download image · 1200 × 630 PNG

Research steps

Adjust for multiple tests

  1. Group outcomes; estimate p-values
  2. Sort and rank within each family
  3. Apply rank × q / M threshold

Illustrative summary of the source-described Benjamini–Hochberg procedure. Use the largest qualifying rank; reject through that rank and report adjusted q-values. No results shown.

See source →

How

When to use

Evaluating workplace interventions across multiple survey outcomes and performance metrics risks false positive findings due to multiple hypothesis testing. Running dozens of individual hypothesis tests increases the probability of erroneously rejecting true null hypotheses by chance alone. Apply false discovery rate control using the Benjamini-Hochberg procedure to adjust p-values and confirm statistical significance across family groups of outcomes.

Adjust p-values across family sets of experimental hypotheses to bound the expected proportion of false positives at level q using the Benjamini-Hochberg procedure.

  1. Group related trial outcomes into coherent hypothesis families, such as productivity metrics, workplace survey outcomes, financial behaviors, and personality traits.
  2. Estimate individual treatment regressions for each outcome variable within a hypothesis family to obtain raw p-values.
  3. Sort the M individual p-values within each family in ascending numerical order.
  4. Assign an integer rank index from one to M to each sorted p-value.
  5. Calculate the false discovery rate critical threshold for each rank by multiplying the rank by target q and dividing by total hypotheses M.
  6. Identify the largest rank where the observed p-value is less than or equal to the calculated critical threshold.
  7. Reject all hypotheses up to that threshold rank and report adjusted false discovery rate q-values for all family outcomes.

Why it works

Grouping outcomes into domain-specific families ensures that multiple testing adjustments evaluate related behavioral constructs together.

Example

Researchers testing soft skills training effects across 27 survey and workplace outcomes grouped hypotheses into six thematic families, including productivity, financial behavior, and personality traits. Applying Benjamini-Hochberg false discovery rate adjustments confirmed that post-program productivity efficiency gains remained statistically significant with a q-value of 0.049 and skill-peer self-assessments remained significant with a q-value of 0.006.

About the study

Participants

2,703 female garment workers across 112 production lines in 5 export factories in Bengaluru, India (621 survey respondents).

Study Design

Two-stage randomized controlled trial using Anderson's implementation of the Benjamini-Hochberg false discovery rate procedure across six outcome families (productivity, workplace survey, financial behavior, entitlements, personality, and mental health).

Results

Multiple testing corrections confirmed post-treatment efficiency gains remained statistically significant (q-value = 0.049; point estimate 0.108), task complexity SAM increases remained significant (q-value = 0.049; point estimate 0.0384), and skill-peer self-assessment remained significant (q-value = 0.006; point estimate 0.158), while extraversion became marginally insignificant (q-value = 0.108).

Source

See source →