Small-Sample Large-T Econometric Experimental Design
Field experiments conducted on large, complex organizations often suffer from severe sample size constraints because recruiting major corporate subjects is difficult and expensive. Standard asymptotic statistical tests fail when the cross-sectional count of firms is small, risking invalid inference or high susceptibility to outlier distortion.
Picture this
Imagine trying to test whether a new training program works by enrolling only 17 sports teams instead of thousands of individual athletes. Because 17 is too small a number to rely on standard statistical averages, researchers collect detailed performance stats on every team every week for two full years, and mathematically shuffle all possible team assignments 12,376 times to ensure that any observed victory was caused by the training and not by luck.
What the evidence says
Intention-To-Treat (ITT) permutation tests demonstrated statistically significant reductions in quality defects (p = 0.001) and output increases (p = 0.026), proving that high-frequency weekly panel data compensates for small cross-sectional sample size.
- Who
- 20 experimental plants across 17 multiplant woven cotton fabric manufacturing firms near Mumbai, India, tracked across ~100 weekly observations per plant.
- How
- Randomized controlled trial combining exact permutation tests across all 12,376 possible treatment assignment combinations with Ibragimov-Mueller (2010) t-statistic procedures that exploit large time-series asymptotics (large T, small N).
What to do
Collect high-frequency repeated measurements over extended time horizons when running interventions on small sample populations to achieve sufficient statistical power and exact inference.
From the source
"To address concerns over statistical inference in small samples, we implemented permutation tests whose properties are independent of sample size. We also exploited our large time series of around 100 weeks of data per plant by using estimators that rely on large T (rather than large N) asymptotics."
541 Management in India QJE.pdf
Tagged
Nearby findings
- Cross-Community Audit Methodology for Bias DetectionEconomics (general)
- Hawthorne Effect Control via Diagnostic-Only Counterfactual DesignEconomics (general)
- Community Incentive and Distance Instrumental Variable MethodEconomics (general)
- Instrumental Variables Correction for Management Measurement ErrorEconomics (general)