Labor Market Information Provision and Job FairsService Sector Female Labor Preference

Evaluation Time Horizon Bias

Download image · 1200 × 630 PNG

In Simple Words

Evaluating a policy right after it starts is like testing a person's diet on day one when they have not had time to buy snacks or change their routine. True long-term policy impacts can only be measured after people have had enough time to notice the change, adjust their habits, and settle into a new equilibrium.

Pupils in the school grant trial

After one year
Test scores increased
After two years
No clear cumulative score gain

Each result compares grant schools with control schools. Expectations were not randomized; household spending is a possible explanation.

See source →

Research question

Standard impact evaluations conducted shortly after program implementation capture temporary short-run effects before economic agents can reoptimize their behavioral decisions. Evaluating interventions over short time horizons leads researchers to confuse transient structural responses with long-run policy impacts.

Participants

200 government primary schools in rural Andhra Pradesh, India (~27,704 student observations in Year 1 vs 19,872 in Year 2).

Study Design

Multi-year randomized evaluation contrasting 1-year short-horizon gains against 2-year long-horizon cumulative gains following household reoptimization.

Results

The estimated intervention effect was 0.085 SD (p < 0.05) after 1 year (before household reoptimization), but fell to a statistically insignificant 0.053 SD (p = 0.611) after 2 years once households adjusted private spending.

Takeaway

Extend impact evaluation timeframes beyond initial implementation periods to allow household and organizational reoptimization before declaring program efficacy.

Source and citation

See source →

Related findings

  • Tangible Goods Simulation for Field BDM Auctions →Methods & evidence

    Before asking someone to place real cash bids on a complex multi-page financial contract, you let them play a practice round bidding 5 cents on a candy bar so they visually see how higher and lower bids win or lose the prize.

  • Birthdate-Based Randomization Design →Methods & evidence

    Assigning people based on their birthdate works like splitting a crowd into two equal groups based on whether their ticket number is odd or even. Because birthdays are evenly distributed across the population, everyone born in the first half of the month gets one treatment and those born in the second half get another, creating two balanced groups without bias.

  • Caste-Stratified Market Penetration Randomization →Methods & evidence

    Imagine testing how a new vaccine affects an entire town's economy depending on what percentage of the population gets vaccinated. Instead of picking some towns to receive 10% vaccination and others 90% (which might accidentally pick wealthier or healthier towns), researchers select specific local community clubs in every town and offer vaccines to members of those chosen clubs.

  • Exogenous Failure Assignment →Methods & evidence

    Imagine testing how drivers handle an unexpected patch of black ice on a road. If drivers only go onto the icy road voluntarily, only risk-tolerant drivers are tested; but if a road detour forces a random selection of drivers onto the ice patch, researchers can measure true handling ability and recovery without selection bias.