Myopic Stopping Rules for Adaptive Field Experiments
RCTClinical Trial
Standard experimental power calculations specify fixed sample sizes upfront, which fails to account for dynamic participant welfare gains or operational implementation costs in adaptive field trials. Without an optimal stopping rule, adaptive trials risk running too long and accumulating redundant implementation expenses or stopping prematurely before identifying optimal treatments.
Picture this
Imagine a company testing new advertisements online. Every day, the company measures how much extra revenue it earns by showing the top ad compared to the daily cost of keeping the test running. As long as the extra earnings and statistical precision gained from another day of testing outweigh the daily operational fee, the test continues; the moment testing costs exceed gains, the trial stops.
What the evidence says
Calculates wave-by-wave net return by summing expected participant welfare net of costs (weighted by treatment assignment shares $(1-\gamma)\hat{p}_{t}^{dx} + \gamma/k$) and the marginal decrease in treatment effect estimator variance. The experiment continues if and only if expected net return is greater than zero.
- Who was studied
- Theoretical framework and empirical decision model applied to trial of N = 3,770 jobseekers in urban Jordan.
- How
- Period-by-period dynamic decision rule evaluating marginal net returns of experimental waves in an adaptive trial governed by Thompson sampling.
What to do
Compute the marginal net return of each experimental wave by balancing expected participant welfare gains and variance reductions against intervention costs, terminating the adaptive trial when net return becomes non-positive.
From the source
"The net return is given by the sum of the weighted average of expected participant welfare net of costs... and the expected decrease of treatment effect estimator variance... Continue the experiment if and only if this expected net return exceeds 0."
An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan