Labor Market Information Provision and Job FairsService Sector Female Labor Preference

Worst-Case Variance Bounded Sampling

Download image · 1200 × 630 PNG

In Simple Words

Think of testing four different types of tires on race cars over a season. If one tire performs poorly early on, you want to stop putting it on drivers' cars to avoid accidents; however, if you stop using it completely, you will not gather enough data to mathematically prove its performance. Setting a strict safety floor guarantees that every tire gets tested on a minimum percentage of cars so statistical proofs remain valid.

Research question

Adaptive multi-armed bandit algorithms dynamically reduce assignment to underperforming treatment arms, which risks severe loss of statistical power and invalidates ex-ante frequentist standard error guarantees.

Participants

Parameterized for N = 4,000 subjects across 16 demographic strata in Jordan.

Study Design

Mathematical derivation establishing an upper bound lambda on estimator variance (lambda = 0.00125) to derive the algorithm tempering parameter gamma.

Results

Setting lambda = 0.00125 guarantees a worst-case standard error of 0.05 for treatment contrasts and a statistical power >= 0.80 for detecting effect sizes of 0.124 across N = 4,000 participants, enforcing a minimum assignment probability floor of 0.05 per treatment arm.

Takeaway

Select the upper variance parameter lambda based on required statistical power and sample size N to establish the minimum treatment assignment share gamma/k.

Source and citation

From the source

"Our experiment has the objective of maximizing participant outcomes... subject to the constraint of allowing for sufficiently precise inference on the effect of different treatments... We require frequentist guarantees on the precision of estimates."

Original source reference

An_Adaptive_Targeted_Field_Experiment_Job_Search_Assistance_for

The original reference is reproduced as recorded. Full publication details have not been independently verified.

Related findings

  • Tangible Goods Simulation for Field BDM Auctions →Methods & evidence

    Before asking someone to place real cash bids on a complex multi-page financial contract, you let them play a practice round bidding 5 cents on a candy bar so they visually see how higher and lower bids win or lose the prize.

  • Birthdate-Based Randomization Design →Methods & evidence

    Assigning people based on their birthdate works like splitting a crowd into two equal groups based on whether their ticket number is odd or even. Because birthdays are evenly distributed across the population, everyone born in the first half of the month gets one treatment and those born in the second half get another, creating two balanced groups without bias.

  • Caste-Stratified Market Penetration Randomization →Methods & evidence

    Imagine testing how a new vaccine affects an entire town's economy depending on what percentage of the population gets vaccinated. Instead of picking some towns to receive 10% vaccination and others 90% (which might accidentally pick wealthier or healthier towns), researchers select specific local community clubs in every town and offer vaccines to members of those chosen clubs.

  • Evaluation Time Horizon Bias →Methods & evidence

    Evaluating a policy right after it starts is like testing a person's diet on day one when they have not had time to buy snacks or change their routine. True long-term policy impacts can only be measured after people have had enough time to notice the change, adjust their habits, and settle into a new equilibrium.