Caseworker-Level Randomization in Policy Evaluation

In Simple Words

Imagine testing a new teaching software in a school. If a single teacher receives the software for half of their students, the teacher will naturally use the insights gained from those students to teach the remaining students, spoiling the experiment. Assigning half the teachers in each school to use the software while the other half teach normally avoids this leak.

Research question

Evaluating decision-support systems by randomizing client individuals creates severe experimental contamination because caseworkers extrapolate predictions learned from treated clients onto control clients. Conversely, randomizing entire employment centers yields too few clusters for valid statistical inference.

Participants

N = 283 regular caseworkers (146 treatment, 137 control) across 21 Swiss regional employment offices, serving a total of N = 35,390 jobseekers (22,758 stock sample, 12,632 flow sample).

Study Design

Cluster Randomized Control Trial with randomization conducted at the individual caseworker level within each employment office, establishing distinct stock and flow samples to isolate historical assignment bias.

Results

Caseworker-level randomization successfully eliminated cross-client informational spillovers while maintaining balanced treatment and control groups across observed jobseeker demographic characteristics (passing balance tests in all regions except Geneva).

Takeaway

Structure field experiments of organizational decision-support tools by randomizing frontline staff rather than end clients to eliminate informational spillovers and cross-treatment extrapolation.

Related findings

• Active Tracking Attrition Mitigation →Methods & evidence

  Dispatching field teams directly to children's homes when they miss school testing operates like a home delivery service tracking down uncollected mail to ensure every package is accounted for.

• Commercial Ad Platform Algorithmic Randomization Imbalance →Methods & evidence

  Allowing a commercial ad algorithm to handle randomization is like letting a profit-driven tournament host pick matching opponents. The platform optimizes for click rates rather than making sure both starting teams are equally balanced.

• Multi-Source Administrative-Survey Data Triangulation →Methods & evidence

  Think of monitoring a patient's health by combining heart rate monitors and blood tests with direct interviews about how they feel. Combining automated medical readings with verbal feedback provides a complete, accurate picture of health rather than relying on either measurement alone.

• Adversarial Non-Stationarity Bounds in Multi-Armed Bandits →Methods & evidence

  Imagine playing a card game where an opponent can secretly alter the deck rules between rounds. If you assume the deck never changes, early luck might trick you into a permanently failing strategy.