Adaptive Randomization Inference
RCTClinical Trial
Standard asymptotic hypothesis testing can introduce severe size distortions and statistical bias when applied to data collected through adaptive assignment algorithms, as assignment probabilities dynamically depend on prior participant outcomes.
Picture this
Imagine testing whether a biased coin affects a game where the rules change every time a player wins. Standard statistical formulas assume the rules stayed fixed throughout the game, leading to incorrect calculations. To fix this, researchers re-run the entire game thousands of times on a computer using the exact same changing rules under the assumption that the coin was fair, creating a benchmark to see how rare the actual outcome really was.
What the evidence says
Delivered exact finite-sample p-values without asymptotic bias; confirmed that 6-week employment gains across all interventions were statistically insignificant (p = 0.296 for cash, p = 0.690 for info, p = 0.388 for nudge), while validating 2-month cash employment gains for Syrian refugees (p = 0.017).
- Who was studied
- N = 3,770 jobseekers (1,663 Syrian refugees and 2,107 Jordanians) in urban Jordan.
- How
- Non-parametric finite-sample randomization inference constructed by repeatedly re-running the Tempered Thompson Algorithm assignment process under the sharp null hypothesis of zero treatment effect across all individuals.
What to do
Construct exact p-values for adaptively generated trial data by re-simulating the adaptive assignment algorithm over repeated permutations of treatment allocations under the sharp null hypothesis.
From the source
"Under this null, we can generate counterfactual data by re-running our assignment algorithm repeatedly, leaving outcomes as they are in our data, but generating new treatment assignments."
An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan