Null findings on placebo questions confirm that respondents are not over-reporting positive actions simply to satisfy survey administrators.

Download image · 1200 × 630 PNG

Research steps

Compare target and placebo survey outcomes

  1. Include indirect and historical placebo questions
  2. Survey while campaign broadcasts remain active
  3. Estimate target and placebo treatment effects

Illustrative summary of the source-described demand-effect checks in Egypt. Placebo questions include resource use before COVID-19, which the intervention cannot change. Compare estimates without treating this check alone as proof against reporting bias.

See source →

How

When to use

In online survey evaluations, respondents might guess study goals and adjust their answers to please researchers. Including placebo questions on uninfluenced outcomes detects whether reported shifts represent real behavioral changes or reporting bias. This method helps researchers evaluating digital interventions on sensitive social issues.

Embed indirect placebo questions into survey instruments to detect potential experimenter demand effects during program evaluation.

  1. Formulate baseline and endline survey instruments containing direct, indirect, and placebo questions.
  2. Select placebo questions measuring historical behaviors that the intervention cannot alter, such as resource use and violence exposure prior to COVID-19.
  3. Frame questions about violence exposure indirectly by asking whether participants or someone they know experienced abuse.
  4. Keep campaign broadcasts active during survey collection so respondents remain unaware that the intervention has ended.
  5. Estimate treatment effects on placebo outcomes alongside primary target outcomes in survey statistical models.
  6. Confirm that placebo outcomes yield precise null results to verify that target outcome shifts reflect true behavioral changes rather than reporting bias.

Why it works

Selective adoption of messaging alongside unchanged historical records validates genuine program impact over social desirability bias.

Example

In Egypt, researchers evaluated endline survey responses from 4,165 women and found precise null effects on placebo outcomes, such as resource use before COVID-19, confirming that observed increases in recent resource use between 0.060 and 0.100 standard deviations were not driven by experimenter demand effects.

About the study

Participants

4,165 female internet users in Egypt who completed the endline survey across five experimental conditions.

Study Design

Randomized controlled trial assessing potential demand bias by comparing treatment impacts on target outcomes against uninfluenced placebo outcomes measuring pre-COVID-19 violence and resource use.

Results

Placebo outcomes showed precise null results across treatment arms (-0.012 to 0.035 effect on pre-COVID-19 resource use), while target outcomes showed statistically significant increases in recent resource use (0.060 to 0.100 standard deviation increase, p < 0.05).

Source

See source →