Try, learn and adjust
Check caseworkers have trial guidance
- Plan: Tell Anna’s agency that trial participation will not affect performance ratings or targets; brief team leaders.
- Try: Train Anna through the team leader using slides, FAQs and a two-sided desk sheet. Keep email and phone support available.
- Learn: Check Anna’s desk guidance and support questions: are assigned trial procedures clear?
- Adjust: Keep guidance if clear. Otherwise change explanations through trial support before the next intake meeting.
Illustrative: Anna is fictional. School’s suggested learning loop, not the source study’s tested protocol.
See source →School’s suggested, untested learning loop. Observe desk guidance and implementation queries. Do not change random assignments or trial arms.
How
When to use
Field experiments in public agencies risk operational resistance if local performance ratings are penalized by experimental protocols. Caseworkers may alter delivery routines if experimental guidelines are unclear or poorly integrated into daily workflows. Insulating agency performance metrics and training caseworkers with standardized desk references ensures consistent implementation during trials.
Train agency caseworkers and insulate local agency ratings before introducing randomized active labor market evaluation protocols.
- Guarantee local agency heads that participating in the evaluation trial will not affect agency performance ratings or targets.
- Present the experimental design to agency leadership and brief caseworker team leaders prior to project launch.
- Instruct single caseworkers through team leaders using standardized presentation slides, FAQ lists, and plastic desk summaries.
- Place two-sided plastic reference sheets summarizing randomized treatment arms on every caseworker's desk throughout the experiment.
- Provide continuous implementation support to caseworkers via email and telephone helplines throughout the trial period.
Why it works
Providing permanent desk reference sheets prevents caseworker confusion and maintains protocol adherence across daily consultation meetings.
Example
In Germany, researchers and federal experts trained team leaders and distributed plastic desk reference sheets across five regional employment agencies. Insulating performance ratings secured full agency cooperation, enabling 213 caseworkers to deliver randomized protocols to 4,163 unemployed men while maintaining baseline balance across groups.
About the study
Participants
213 caseworkers across 5 regional employment agencies in Germany, serving 4,163 male job seekers aged 25 to 64.
Study Design
Organizational training protocol and performance goal insulation evaluated alongside individual-level randomized controlled trial protocols.
Results
F-tests across baseline characteristics confirmed robust protocol compliance and group balance (overall multinomial logit balancing p-value = 0.42) across all 213 caseworkers, while treatment effects varied significantly across caseworkers for low-employability job seekers (F-test p-value = 0.041).