Policy-Relevant Treatment Effect vs. Microeconometric Bias

In Simple Words

Testing a free express toll pass on a tiny fraction of drivers makes those drivers arrive much faster without slowing down anyone else. However, giving the pass to every driver on the road creates massive gridlock in the express lane, completely eliminating the original time savings.

Research question

Standard microeconometric evaluations evaluate randomized trials at a single observed treatment intensity, assuming individual outcomes are independent of peer treatment status. This creates substantial evaluation bias when predicting the net benefits of nationwide policy implementations.

Participants

N = 89,466 benefit spells evaluated across experimental ($\tau = 0.3$) and full rollout ($\tau = 1.0$) conditions in Denmark.

Study Design

Comparison of experimental difference-in-means estimates against general equilibrium policy-relevant treatment effect simulations.

Results

At experimental intensity ($\tau = 0.3$), participants enjoyed a 5.4 percentage point higher monthly matching rate than nonparticipants (0.245 vs 0.191). However, the policy-relevant treatment effect under full rollout ($\tau = 1.0$) was only 0.3 percentage points (0.208 vs 0.205), proving standard microeconometric trials overestimate nationwide impact by over 1,700%.

Takeaway

Distinguish microeconometric trial treatment effects from policy-relevant general equilibrium treatment effects before scaling localized experimental programs.

Read alongside the research

Plain-English glossary

Original explanations, informed by the references below. Examples are illustrations, not data from the studies above. The research text stays unchanged.

Percentage points

The difference between two percentages.

ExampleA rise from 40% to 50% is 10 percentage points. Relative to the starting value, that is a 25% increase.

Office for National Statistics · Percentages ↗Read more about percentage points →

Related findings

• Cultivator Risk-Taking and Labor Demand Volatility →Economics (general)

  Imagine a factory owner who usually manufactures simple wooden chairs because wooden chairs do not break if the factory roof leaks during a storm. If an insurance company guarantees to pay the factory owner every time a storm leaks through the roof, the owner immediately switches to building high-end electronic gadgets that produce huge profits during dry weather but get ruined in a rainstorm.

• Dual-Market Side Insurance Neutrality →Economics (general)

  Think of a tug-of-war game between store owners who want to cut wages during slow periods and shop assistants who demand higher pay. If only store owners receive insurance, they aggressively expand or contract hiring based on the weather, causing wild wage swings.

• General Equilibrium Program Evaluation Gap in Scaled Interventions →Economics (general)

  Imagine giving discount grocery vouchers to half the families in an isolated village. A standard test only checks if those receiving vouchers eat better, but when half the village uses vouchers to buy more food, the local store owner raises prices for everyone.

• Insurable Interest Exclusion and GE Welfare Spillover →Economics (general)

  Imagine a law that allows shop owners in a town to buy fire insurance, but forbids the shop employees from buying unemployment insurance. When shop owners buy fire insurance, they stock their shelves with fragile glass statues that yield huge profits when safe, but shatter instantly during fires.