Unannounced Paper-and-Pencil Survey Measurement Protocol
RCTClinical Trial
Evaluating digital interventions using computer-based surveys can introduce measurement bias, as treatment participants gain greater computer familiarity than control participants during the intervention.
Picture this
Instead of testing students on the same computers used for lessons—which would give tech-trained students an unfair advantage—evaluators hand out unexpected physical paper tests in class. This ensures everyone answers under identical physical conditions regardless of their computer skill.
What the evidence says
Attrition was limited to 13% between baseline and first follow-up, and 10% between baseline and second follow-up, with no differential attrition across treatment and control arms.
- Who was studied
- N = 4,599 human ninth-grade public school students across 138 classrooms in 21 Colombian cities.
- How
- Three unannounced rounds of paper-and-pencil surveys (baseline, post-course at 1 week, and 6-month follow-up) to minimize familiarization bias and attrition.
What to do
Administer offline paper-and-pencil evaluations when testing computer-based educational programs to eliminate modality familiarity bias.
From the source
"The pencil and paper survey strategy was chosen instead of a computer survey because treatment students would have more familiarity with computers at endline, hence unnecessarily biasing results."
Do Information Technologies Improve Teenagers’ Sexual Education? Evidence from a Randomized Evaluation in Colombia