Expert Forecast Bias and Overoptimism

Picture this

Think of asking seasoned local coaches to guess how much a new training drill will boost player scoring before a game. If local coaches predict a massive 20% scoring increase, while the actual improvement is under 1%, relying on intuition causes managers to overinvest in ineffective drills. Eliciting incentivized numerical forecasts before launching interventions exposes baseline prediction errors and uncertainty.

What the evidence says

Local staff exhibited substantial overoptimism, predicting median 6-week employment rates of 20% for cash, 10% for information, and 9% for nudge (against actual impact under 1 percentage point). Senior staff predictions were significantly more accurate (7% cash, 5% info, 4% nudge).

Who was studied

N = 20 program experts (16 local implementation staff in Amman, 4 senior international directors in New York).

How

Incentivized online forecast elicitation asking experts to predict 6-week post-treatment employment rates for cash, information, and nudge interventions against a 2.5% baseline.

What to do

Administer incentivized pre-trial forecast surveys to implementation staff to establish prior belief distributions and quantify prediction biases before rolling out field interventions.

From the source

"Relative to our estimated treatment effects, local staff were very optimistic in their forecasts: at the median, they predicted employment rates of 20%, 10% and 9% for the cash, information and nudge interventions... Senior staff had more accurate forecasts: medians of 7%, 5% and 4%."

An_Adaptive_Targeted_Field_Experiment_Job_Search_Assistance_for.pdf

Tags

• expert forecasts
• policy design
• overoptimism