Caseworker-Level Randomization in Policy Evaluation
Evaluating decision-support systems by randomizing client individuals creates severe experimental contamination because caseworkers extrapolate predictions learned from treated clients onto control clients. Conversely, randomizing entire employment centers yields too few clusters for valid statistical inference.
Picture this
Imagine testing a new teaching software in a school. If a single teacher receives the software for half of their students, the teacher will naturally use the insights gained from those students to teach the remaining students, spoiling the experiment. Assigning half the teachers in each school to use the software while the other half teach normally avoids this leak.
What the evidence says
Caseworker-level randomization successfully eliminated cross-client informational spillovers while maintaining balanced treatment and control groups across observed jobseeker demographic characteristics (passing balance tests in all regions except Geneva).
- Who
- N = 283 regular caseworkers (146 treatment, 137 control) across 21 Swiss regional employment offices, serving a total of N = 35,390 jobseekers (22,758 stock sample, 12,632 flow sample).
- How
- Cluster Randomized Control Trial with randomization conducted at the individual caseworker level within each employment office, establishing distinct stock and flow samples to isolate historical assignment bias.
What to do
Structure field experiments of organizational decision-support tools by randomizing frontline staff rather than end clients to eliminate informational spillovers and cross-treatment extrapolation.
From the source
"A randomization at the level of the caseworker was preferred to a randomization at the level of the employment office or at the level of the unemployed person... A randomization at the level of the unemployed person, on the other hand, would have led to the problem that a caseworker would receive employment predictions for some of his clients but not for others... produce spill over effects."
Targeting_Labour_Market_Programmes_Results_from_a_Randomized_Experiment.pdf