Expert Forecast Overoptimism
RCTClinical Trial
Policy interventions and development programs are often selected based on subjective expert opinions, but experts may hold overly optimistic or inaccurate beliefs regarding treatment effectiveness, leading to suboptimal allocation of humanitarian funds.
Picture this
Imagine asking local basketball coaches to predict how many games a new training routine will win before the season starts. Because the coaches are excited about the program, they predict the team will win 20 games, whereas actual match results show the team only wins 1 game. Eliciting predictions before the season starts allows managers to measure how far off expert intuition is from actual performance and make better decisions in the future.
What the evidence says
Local field staff exhibited substantial overoptimism, predicting median 6-week employment rates of 20% for cash, 10% for information, and 9% for nudge (compared to actual control/treatment rates under 5%). Senior international staff had more accurate median forecasts (7% for cash, 5% for info, 4% for nudge). Both groups correctly anticipated that cash would be the most effective intervention.
- Who was studied
- N = 20 implementation experts (16 local employment staff in Amman, Jordan, and 4 senior international management staff in New York).
- How
- Online incentivized belief elicitation exercise measuring predicted 6-week wage employment rates across control and three treatment arms prior to field trial completion.
What to do
Elicit incentivized baseline outcome predictions from implementation staff prior to intervention trials to quantify professional expectation biases and calibrate Bayesian priors.
From the source
"Relative to our estimated treatment effects, local staff were very optimistic in their forecasts: at the median, they predicted employment rates of 20%, 10% and 9% for the cash, information and nudge interventions respectively..."
An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan