Evaluation Time Horizon Bias
Standard impact evaluations conducted shortly after program implementation capture temporary short-run effects before economic agents can reoptimize their behavioral decisions. Evaluating interventions over short time horizons leads researchers to confuse transient structural responses with long-run policy impacts.
Picture this
Evaluating a policy right after it starts is like testing a person's diet on day one when they have not had time to buy snacks or change their routine. True long-term policy impacts can only be measured after people have had enough time to notice the change, adjust their habits, and settle into a new equilibrium.
What the evidence says
The estimated intervention effect was 0.085 SD (p < 0.05) after 1 year (before household reoptimization), but fell to a statistically insignificant 0.053 SD (p = 0.611) after 2 years once households adjusted private spending.
- Who
- 200 government primary schools in rural Andhra Pradesh, India (~27,704 student observations in Year 1 vs 19,872 in Year 2).
- How
- Multi-year randomized evaluation contrasting 1-year short-horizon gains against 2-year long-horizon cumulative gains following household reoptimization.
What to do
Extend impact evaluation timeframes beyond initial implementation periods to allow household and organizational reoptimization before declaring program efficacy.
From the source
"...the interpretation of experimental coefficients depends on the time horizon of the evaluation and whether this was long enough for other agents to reoptimize their own inputs."
School Inputs, Household Substitution, and Test Scores