Randomization Inference Permutation Testing

In Simple Words

Think of shuffling a deck of cards thousands of times to see how often a rare winning hand appears purely by random chance. By randomly swapping treatment labels across participants 10,000 times, researchers construct an exact benchmark distribution to verify if the real experimental result was a true effect or a statistical fluke.

Research question

Standard asymptotic t-tests in randomized field experiments can produce overconfident significance claims when outcome distributions exhibit heavy skewness, clustering, or potential outlier distortion.

Participants

N = 7,438 establishments across 129 local public employment agencies in France.

Study Design

Randomization inference student test using 10,000 permutation runs within assignment strata (Young, 2018) alongside Mann-Whitney ranksum tests.

Results

Permutation testing yielded exact p-values of p = 0.024 for permanent hires of registered jobseekers and p = 0.037 for theoretical workdays created, confirming significant treatment effects independently of asymptotic distributional assumptions.

Takeaway

Validate field experiment statistical significance by running 10,000-iteration randomization permutation tests alongside standard asymptotic t-tests.

Related findings

• Counterfactual Re-Randomization Inference →Economics (general)

  Imagine evaluating whether a coin is biased during a game where the referee alters the flipping rules every time someone scores. To test fairly if players had any real effect on the game, you replay the exact sequence of changing referee rules thousands of times assuming player actions mattered zero percent, creating a true benchmark to see how rare the real game result actually was.

• Administrative Social Security Data Linkage for Earnings Measurement →Economics (general)

  Instead of asking individuals to remember and report exact yearly income numbers from a decade ago, researchers obtain signed consent to link survey profiles directly to official government tax records. This creates an unalterable, objective earnings ledger covering years before and after a financial event.

• Agnostic Machine Learning Heterogeneity Detection in Field Trials →Economics (general)

  Evaluating a program using only average results is like measuring the average temperature of a room with one frozen wall and one burning wall and concluding the room is comfortable. Machine learning acts like a thermal camera, splitting the room into sections to reveal the extreme cold and extreme heat hiding behind the average.

• Asymmetric Wealth Effect of Subsidized Index Contracts →Economics (general)

  Imagine buying a lottery ticket that costs $1, but pays out $1,000 if a major storm hits. Paying $1 during normal weather is so cheap that it does not change weekly spending or force extra work hours to make up for the cost.