Frequentist Variance Bounding in Adaptive Trials

In Simple Words

Imagine an adaptive slot machine experiment where the casino wants to stop players from using losing machines. If the casino completely turns off losing machines after a few bad spins, researchers can never prove mathematically how bad those machines actually were. To solve this, the casino sets a mandatory rule: every machine must stay active at least 5% of the time, guaranteeing enough statistical data to prove exact performance while still sending 80% of players to the winning machine.

Research question

Pure adaptive multi-armed bandit algorithms can reduce assignment probabilities of suboptimal treatment arms to near zero, causing severe variance inflation, statistical power collapse, and distorted frequentist inference.

Participants

N = 3,770 jobseekers in urban Jordan.

Study Design

Theoretical derivation and empirical implementation of a frequentist variance bound $\lambda = 0.00125$, setting minimum assignment share $\overline{q}^{dx} \ge \frac{1}{4 N \lambda} = 0.05$ across $k=4$ arms.

Results

Bounded the worst-case standard error of estimated average treatment effects at $\sqrt{2\lambda} = 0.05$ (equivalent to 80% power for detecting effect sizes of 0.124), resulting in an actual empirical standard error bound of 0.016 in the trial.

Takeaway

Set a minimum assignment share parameter $\gamma = \frac{1}{4 N \lambda}$ in multi-armed bandit rules to enforce a frequentist bound $\lambda$ on estimator variance across all treatment arms.

Related findings

• Counterfactual Re-Randomization Inference →Economics (general)

  Imagine evaluating whether a coin is biased during a game where the referee alters the flipping rules every time someone scores. To test fairly if players had any real effect on the game, you replay the exact sequence of changing referee rules thousands of times assuming player actions mattered zero percent, creating a true benchmark to see how rare the real game result actually was.

• Myopic Stopping Rules for Adaptive Field Experiments →Economics (general)

  Imagine a company testing new advertisements online. Every day, the company measures how much extra revenue it earns by showing the top ad compared to the daily cost of keeping the test running.

• Statistical Surrogacy in Sequential Policy Trials →Economics (general)

  Imagine a school testing three teaching methods to improve student graduation rates ten years later. Waiting a decade to update the teaching strategy makes adaptive learning impossible.

• Tempered Thompson Algorithm →Economics (general)

  Imagine a doctor testing four different medical treatments on arriving patients. Instead of flipping a fair four-sided coin for every single patient throughout the entire study, the doctor tracks which treatment is working best.