Research steps
Compare target and placebo survey outcomes
- Include indirect and historical placebo questions
- Survey while campaign broadcasts remain active
- Estimate target and placebo treatment effects
Illustrative summary of the source-described demand-effect checks in Egypt. Placebo questions include resource use before COVID-19, which the intervention cannot change. Compare estimates without treating this check alone as proof against reporting bias.
See source →How
When to use
In online survey evaluations, respondents might guess study goals and adjust their answers to please researchers. Including placebo questions on uninfluenced outcomes detects whether reported shifts represent real behavioral changes or reporting bias. This method helps researchers evaluating digital interventions on sensitive social issues.
Embed indirect placebo questions into survey instruments to detect potential experimenter demand effects during program evaluation.
- Formulate baseline and endline survey instruments containing direct, indirect, and placebo questions.
- Select placebo questions measuring historical behaviors that the intervention cannot alter, such as resource use and violence exposure prior to COVID-19.
- Frame questions about violence exposure indirectly by asking whether participants or someone they know experienced abuse.
- Keep campaign broadcasts active during survey collection so respondents remain unaware that the intervention has ended.
- Estimate treatment effects on placebo outcomes alongside primary target outcomes in survey statistical models.
- Confirm that placebo outcomes yield precise null results to verify that target outcome shifts reflect true behavioral changes rather than reporting bias.
Why it works
Selective adoption of messaging alongside unchanged historical records validates genuine program impact over social desirability bias.
Example
In Egypt, researchers evaluated endline survey responses from 4,165 women and found precise null effects on placebo outcomes, such as resource use before COVID-19, confirming that observed increases in recent resource use between 0.060 and 0.100 standard deviations were not driven by experimenter demand effects.
About the study
Participants
4,165 female internet users in Egypt who completed the endline survey across five experimental conditions.
Study Design
Randomized controlled trial assessing potential demand bias by comparing treatment impacts on target outcomes against uninfluenced placebo outcomes measuring pre-COVID-19 violence and resource use.
Results
Placebo outcomes showed precise null results across treatment arms (-0.012 to 0.035 effect on pre-COVID-19 resource use), while target outcomes showed statistically significant increases in recent resource use (0.060 to 0.100 standard deviation increase, p < 0.05).