Source · aikyam school

An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan

Original source

A. Stefano Caria; Grant Gordon; Maximilian Kasy; Simon Quinn; Soha Shami; Alexander Teytelboym

An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan

Preprint · 4 May 2023

Search for this source ↗

Permission for School’s use is unestablished.

Used in these cards

Maintain non-zero minimum treatment assignment probabilities ($\gamma > 0$) in adaptive algorithms to secure $O(1/\sqrt{T})$ worst-case regret bounds against non-stationary shocks during field deployment.

"Remarkably, bandit algorithms (such as Thompson sampling) are able to achieve worst-case average regret (relative to any fixed treatment or targeted assignment policy) on the order of $1/\sqrt{T}$ (up to logarithmic terms)."

Elicit incentivized baseline outcome predictions from implementation staff prior to intervention trials to quantify professional expectation biases and calibrate Bayesian priors.

"Relative to our estimated treatment effects, local staff were very optimistic in their forecasts: at the median, they predicted employment rates of 20%, 10% and 9% for the cash, information and nudge interventions respectively..."

Set a minimum assignment share parameter $\gamma = \frac{1}{4 N \lambda}$ in multi-armed bandit rules to enforce a frequentist bound $\lambda$ on estimator variance across all treatment arms.

"Our frequentist (i.e., ex-ante, worst-case) constraint in the experiment is that $max_x max_{\theta_0} p^x Var_{\theta_0}(\overline{Y}^{dx}) \le \lambda$... This allows the designer to target participant welfare while ensuring that she can learn something about the effectiveness of suboptimal treatments."

Deploy structured weekly planning calendars paired with automated goal-setting and goal-review text messages to jobseekers to mitigate self-control bottlenecks during active search phases.

"Through the calendar and the SMSs, participants track the number of jobs they intend to apply for and hours of job search they intend to spend, and can then check whether they met their goals for the week."

Evaluate baseline search intensity and liquidity metrics separately for host national and refugee populations before deploying active labor market assistance to avoid misallocating funds.

"We find essentially no positive effects of treatment on the Jordanian sample. The Syrians and Jordanians who take part in our study differ on multiple dimensions at baseline, and their outcomes evolve in different ways over time."

Negotiate international compacts that combine trade concessions and concessional financing for host nations with formal commitments to grant work permits and reduce regulatory labor barriers for displaced populations.

"In exchange for trade concessions and access to conditional financing, the Government of Jordan committed to important policy changes aimed at improving access of Syrian refugees to the labor market."
"As a result, four months after treatment, the grant boosts employment by 3.8 percentage points (70 percent) and earnings by 65 percent."

Compute the marginal net return of each experimental wave by balancing expected participant welfare gains and variance reductions against intervention costs, terminating the adaptive trial when net return becomes non-positive.

"The net return is given by the sum of the weighted average of expected participant welfare net of costs... and the expected decrease of treatment effect estimator variance... Continue the experiment if and only if this expected net return exceeds 0."

Validate short-run outcomes against the surrogacy condition before deploying them as reward metrics in adaptive assignment algorithms to ensure alignment with long-term policy welfare.

"This condition requires that the long-term outcome should be independent of the treatment assignment conditional on observable characteristics X and short-run outcomes."

Set the mixing parameter gamma in a multi-armed bandit algorithm to enforce a lower bound gamma/k on assignment probabilities for all k treatment arms to balance participant welfare and frequentist inference bounds.

"Our Tempered Thompson Algorithm combines the estimation objective of conventional RCTs with the welfare-maximizing objective of bandit algorithms."

Employ familiar, trusted field staff rather than anonymous survey agents to administer follow-up evaluations when measuring informal labor market outcomes among vulnerable populations.

"ESO surveys were conducted by staff members familiar to the respondents, whereas the rapid-response surveys were conducted by someone whom the respondents had not previously met. It is thus likely that many respondents, and particularly Syrian refugees, had more trust in the ESO enumerators, and hence were more likely to discuss informal work with them."

Calculate formal welfare contrasts ($\Delta_1, \Delta_2, \Delta_3$) comparing observed adaptive allocations, optimal targeted policies, and non-targeted policies against baseline controls to isolate targeting gains from adaptive design gains.

"We estimate that the optimal targeted policy has a treatment effect on six-week employment that is one percentage point larger than the optimal non-targeted policy..."