Policy-Relevant Treatment Effect vs. Microeconometric Bias
Standard microeconometric evaluations evaluate randomized trials at a single observed treatment intensity, assuming individual outcomes are independent of peer treatment status. This creates substantial evaluation bias when predicting the net benefits of nationwide policy implementations.
Picture this
Testing a free express toll pass on a tiny fraction of drivers makes those drivers arrive much faster without slowing down anyone else. However, giving the pass to every driver on the road creates massive gridlock in the express lane, completely eliminating the original time savings.
What the evidence says
At experimental intensity ($\tau = 0.3$), participants enjoyed a 5.4 percentage point higher monthly matching rate than nonparticipants (0.245 vs 0.191). However, the policy-relevant treatment effect under full rollout ($\tau = 1.0$) was only 0.3 percentage points (0.208 vs 0.205), proving standard microeconometric trials overestimate nationwide impact by over 1,700%.
- Who
- N = 89,466 benefit spells evaluated across experimental ($\tau = 0.3$) and full rollout ($\tau = 1.0$) conditions in Denmark.
- How
- Comparison of experimental difference-in-means estimates against general equilibrium policy-relevant treatment effect simulations.
What to do
Distinguish microeconometric trial treatment effects from policy-relevant general equilibrium treatment effects before scaling localized experimental programs.
From the source
"The evaluation of the randomized experiment estimates a treatment effect on the matching rates equal to 0.245-0.191=0.054... However, as we mentioned before the policy relevant treatment effect is E[m(a1*; a, θ)|τ=1]-E[m(a0*; a, θ)|τ=0], which is 0.208-0.205=0.003."
Estimating Equilibrium Effects of Job Search Assistance