Pre-Experimental Out-of-Sample Predictive Profiling

In Simple Words

Imagine testing a new teaching method on current students. Instead of grouping students using current test scores—which might be altered by the study itself or small sample noise—a formula built on last year's student records is used to categorize current students before measuring results.

Research question

In randomized evaluations, dividing experimental subjects using endogenously estimated control group models introduces sample overfitting and systematic treatment effect estimation biases.

Participants

N = 55,545 male UI entrants from 2011 (historical pre-experimental cohort) used to classify N = 4,163 male job seekers aged 25–64 in Germany.

Study Design

Out-of-sample Weibull Proportional Hazard duration model estimated on historical 2011 administrative register data to predict individual median unemployment duration ($m(T|x)$) without using experimental sample outcomes.

Results

The historical model successfully stratified the experimental sample into 1,688 high-employability ($\le 6$ months) and 2,475 low-employability ($> 6$ months) individuals, correctly identifying 87% of long-duration cases in baseline data without endogenous stratification bias.

Takeaway

Estimate baseline duration and profiling models on independent pre-experimental historical cohorts rather than in-sample experimental controls when evaluating subgroup treatment heterogeneity.

Read alongside the research

Plain-English glossary

Original explanations, informed by the references below. Examples are illustrations, not data from the studies above. The research text stays unchanged.

Median

The middle number after arranging the values from smallest to largest. If there are two middle numbers, use their average.

ExampleFor the values 2, 4 and 12, the middle value is 4. The median is 4.

NIST · Measures of location ↗Read more about median →

Baseline

The starting measurements in a study. Researchers use these to compare what participants were like at the start with what happens later.

ExampleBefore a training programme starts, researchers measure workers’ earnings. That first measurement is the baseline.

GET-IT glossary · Baseline characteristics ↗Read more about baseline →

Related findings

• Ability-Dependent Peer Screening →Labour & employment

  Imagine trying to pick the best chess player out of a crowd. Only a master chess player has the expertise to accurately assess who among their acquaintances truly plays at an elite level, whereas a beginner cannot distinguish between a decent amateur and a grandmaster.

• Ability-Heterogeneous Wage Returns →Labour & employment

  Imagine a nightclub with a strict dress code based on exam scores where high scorers enter easily without extra effort. For high-scoring individuals, additional work experience adds little value because employers already hire them based on test scores.

• Activation Program Threat Effect →Labour & employment

  Think of a mandatory weekend study hall for students behind on homework. Many students work much harder during the week to finish their assignments on time simply to avoid having to show up for the study hall in the first place.

• Actual vs. Potential Experience Bias →Labour & employment

  Calculating how many years someone has been a driver by subtracting their high school graduation year from their current age creates significant error if they spent five years without a car. Taking a simple formula creates a false picture of driving practice.