Frequentist Variance Bounding in Adaptive Trials
RCTClinical Trial
Pure adaptive multi-armed bandit algorithms can reduce assignment probabilities of suboptimal treatment arms to near zero, causing severe variance inflation, statistical power collapse, and distorted frequentist inference.
Picture this
Imagine an adaptive slot machine experiment where the casino wants to stop players from using losing machines. If the casino completely turns off losing machines after a few bad spins, researchers can never prove mathematically how bad those machines actually were. To solve this, the casino sets a mandatory rule: every machine must stay active at least 5% of the time, guaranteeing enough statistical data to prove exact performance while still sending 80% of players to the winning machine.
What the evidence says
Bounded the worst-case standard error of estimated average treatment effects at $\sqrt{2\lambda} = 0.05$ (equivalent to 80% power for detecting effect sizes of 0.124), resulting in an actual empirical standard error bound of 0.016 in the trial.
- Who was studied
- N = 3,770 jobseekers in urban Jordan.
- How
- Theoretical derivation and empirical implementation of a frequentist variance bound $\lambda = 0.00125$, setting minimum assignment share $\overline{q}^{dx} \ge \frac{1}{4 N \lambda} = 0.05$ across $k=4$ arms.
What to do
Set a minimum assignment share parameter $\gamma = \frac{1}{4 N \lambda}$ in multi-armed bandit rules to enforce a frequentist bound $\lambda$ on estimator variance across all treatment arms.
From the source
"Our frequentist (i.e., ex-ante, worst-case) constraint in the experiment is that $max_x max_{\theta_0} p^x Var_{\theta_0}(\overline{Y}^{dx}) \le \lambda$... This allows the designer to target participant welfare while ensuring that she can learn something about the effectiveness of suboptimal treatments."
An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan