Restricting analysis to active platform users in both arms compares treated individuals only against control subjects who would have received treatment had their cluster been assigned to it.

Download image · 1200 × 630 PNG

Research design

Compare active users in both study arms

After the campaign, ask about platform visits in the preceding week

Assigned treatment
Retain respondents reporting platform visits; exclude zero visits.
Assigned control
Retain respondents reporting platform visits; exclude zero visits.

Illustrative summary of the source’s symmetric user restriction. Compare outcomes among retained active users and calculate confidence intervals accounting for blocked, clustered assignment. Control users would have been eligible for ads under treatment assignment; no effects are shown.

See source →

How

When to use

Researchers evaluating online campaigns need to estimate ad effects specifically among individuals actually exposed to the media. Overall comparisons include non-users who never log in and could not view the ads, diluting treatment estimates. Filtering both treatment and control samples down to active platform users isolates the treatment-on-treated effect without introducing non-compliance bias.

Measure treatment effects among platform users by restricting analysis to respondents in both arms who logged into the site during the campaign period.

  1. Survey respondents in both randomly assigned treatment and control groups following the ad deployment window.
  2. Ask respondents how frequently they logged onto the specific advertising platform over the preceding week.
  3. Exclude survey respondents from both treatment and control groups who report zero platform visits during the treatment week.
  4. Define the remaining sub-sample as active platform users who were exposed in the treatment group or would have been exposed in the control group.
  5. Estimate treatment-on-treated effects by comparing survey outcomes between treatment and control respondents within this active user sub-sample.
  6. Calculate confidence intervals taking into account the blocked and clustered randomization scheme.

Why it works

This placebo-style restriction removes non-users who could never receive treatment, providing an unbiased estimate of the treatment effect on actual viewers.

Example

In Study 1, researchers restricted analysis to 1,364 respondents who reported visiting Facebook during the campaign week out of 2,984 total survey respondents, measuring the effect of ad exposure among actual platform visitors across treatment and control clusters.

About the study

Participants

1,364 self-reported Facebook users out of 2,984 survey respondents in Study 1, and 1,337 Facebook users out of 3,085 survey respondents in Study 2.

Study Design

Placebo-controlled sub-sample analysis comparing active Facebook users in assigned treatment clusters to active Facebook users in assigned control clusters.

Results

In Study 1, the treatment-on-treated effect among Facebook users was -1.2 percentage points for candidate recognition (95% CI [-0.056, 0.033]) and -0.3 percentage points for favorability (95% CI [-0.041, 0.037]); in Study 2, Facebook users showed an 8.1 percentage point increase in online ad recall (95% CI [0.035, 0.125], p < 0.001) but non-significant shifts in candidate recognition (-0.8 percentage points, 95% CI [-0.067, 0.053]) and favorability (+1.1 percentage points, 95% CI [-0.045, 0.072]).

Source

See source →