Counterfactual Re-Randomization Inference
Expert TheoryReview
Data-dependent allocation algorithms change assignment probabilities over time, breaking standard independent sampling assumptions and risking biased p-values or size distortions in finite-sample inference.
Picture this
Imagine evaluating whether a coin is biased during a game where the referee alters the flipping rules every time someone scores. To test fairly if players had any real effect on the game, you replay the exact sequence of changing referee rules thousands of times assuming player actions mattered zero percent, creating a true benchmark to see how rare the real game result actually was.
What the evidence says
Generated exact finite-sample p-values controlling for time-varying allocation history without relying on asymptotic normality, verifying that short-term 6-week employment differences across all treatment arms had p-values well above conventional significance thresholds (p > 0.12).
- Who was studied
- Evaluated across N = 3,770 jobseekers across 16 demographic strata in Jordan.
- How
- Finite-sample non-parametric inference under the sharp null hypothesis ($Y_{it}^d = Y_{it}^{d'}$) by re-simulating the time-varying adaptive assignment algorithm repeatedly while keeping observed outcomes fixed.
What to do
Re-run adaptive allocation algorithms over observed experimental histories under the sharp null hypothesis to generate exact finite-sample p-values.
From the source
"Under this null, we can generate counterfactual data by re-running our assignment algorithm repeatedly, leaving outcomes as they are in our data, but generating new treatment assignments. The distribution of test-statistics over this re-randomization distribution can be used to construct critical values and p-values that are exact in finite samples..."
An_Adaptive_Targeted_Field_Experiment_Job_Search_Assistance_for.pdf