Research design
Compare active users in both study arms
After the campaign, ask about platform visits in the preceding week
- Assigned treatment
- Retain respondents reporting platform visits; exclude zero visits.
- Assigned control
- Retain respondents reporting platform visits; exclude zero visits.
Illustrative summary of the source’s symmetric user restriction. Compare outcomes among retained active users and calculate confidence intervals accounting for blocked, clustered assignment. Control users would have been eligible for ads under treatment assignment; no effects are shown.
See source →How
When to use
Researchers evaluating online campaigns need to estimate ad effects specifically among individuals actually exposed to the media. Overall comparisons include non-users who never log in and could not view the ads, diluting treatment estimates. Filtering both treatment and control samples down to active platform users isolates the treatment-on-treated effect without introducing non-compliance bias.
Measure treatment effects among platform users by restricting analysis to respondents in both arms who logged into the site during the campaign period.
- Survey respondents in both randomly assigned treatment and control groups following the ad deployment window.
- Ask respondents how frequently they logged onto the specific advertising platform over the preceding week.
- Exclude survey respondents from both treatment and control groups who report zero platform visits during the treatment week.
- Define the remaining sub-sample as active platform users who were exposed in the treatment group or would have been exposed in the control group.
- Estimate treatment-on-treated effects by comparing survey outcomes between treatment and control respondents within this active user sub-sample.
- Calculate confidence intervals taking into account the blocked and clustered randomization scheme.
Why it works
This placebo-style restriction removes non-users who could never receive treatment, providing an unbiased estimate of the treatment effect on actual viewers.
Example
In Study 1, researchers restricted analysis to 1,364 respondents who reported visiting Facebook during the campaign week out of 2,984 total survey respondents, measuring the effect of ad exposure among actual platform visitors across treatment and control clusters.
About the study
Participants
1,364 self-reported Facebook users out of 2,984 survey respondents in Study 1, and 1,337 Facebook users out of 3,085 survey respondents in Study 2.
Study Design
Placebo-controlled sub-sample analysis comparing active Facebook users in assigned treatment clusters to active Facebook users in assigned control clusters.
Results
In Study 1, the treatment-on-treated effect among Facebook users was -1.2 percentage points for candidate recognition (95% CI [-0.056, 0.033]) and -0.3 percentage points for favorability (95% CI [-0.041, 0.037]); in Study 2, Facebook users showed an 8.1 percentage point increase in online ad recall (95% CI [0.035, 0.125], p < 0.001) but non-significant shifts in candidate recognition (-0.8 percentage points, 95% CI [-0.067, 0.053]) and favorability (+1.1 percentage points, 95% CI [-0.045, 0.072]).