Non-Applicant Selection Bias in Active Labor Market Evaluations

In Simple Words

Evaluating a program using only people who signed up is like judging the popularity of a movie by surveying people inside the theater, while completely ignoring everyone who looked at the movie poster outside and walked away.

Research question

Standard randomized evaluations of job training programs estimate treatment impacts by comparing accepted applicants with rejected applicants, systematically ignoring eligible non-applicants who were deterred during recruitment.

Participants

N = 1,460 youth in Experiment 1 and N = 768 youth in Experiment 2 in Cairo, Egypt.

Study Design

Randomized field trial analyzing recruitment-stage eligibility and application drop-off before post-application randomization.

Results

Control application rate was 43% in Experiment 1 and job fair attendance was 5.9% in Experiment 2, establishing that recruitment messaging alters who enters the sample pool before program training impact evaluation occurs.

Takeaway

Measure recruitment-stage eligibility and non-applicant drop-off when evaluating active labor market policies rather than restricting randomization to existing applicant pools.

Related findings

• General Equilibrium Program Evaluation Gap in Scaled Interventions →Economics (general)

  Imagine giving discount grocery vouchers to half the families in an isolated village. A standard test only checks if those receiving vouchers eat better, but when half the village uses vouchers to buy more food, the local store owner raises prices for everyone.

• Monotonic Selection Bounds Adjustment →Economics (general)

  Think of a running coach whose new training program keeps slower runners from quitting the team. If team speed is measured without adjusting for the fact that slower runners now participate in time trials, overall average team speed appears lower than it actually is.

• Administrative Social Security Data Linkage for Earnings Measurement →Economics (general)

  Instead of asking individuals to remember and report exact yearly income numbers from a decade ago, researchers obtain signed consent to link survey profiles directly to official government tax records. This creates an unalterable, objective earnings ledger covering years before and after a financial event.

• Agnostic Machine Learning Heterogeneity Detection in Field Trials →Economics (general)

  Evaluating a program using only average results is like measuring the average temperature of a room with one frozen wall and one burning wall and concluding the room is comfortable. Machine learning acts like a thermal camera, splitting the room into sections to reveal the extreme cold and extreme heat hiding behind the average.