Unannounced Paper-and-Pencil Survey Measurement Protocol

In Simple Words

Instead of testing students on the same computers used for lessons—which would give tech-trained students an unfair advantage—evaluators hand out unexpected physical paper tests in class. This ensures everyone answers under identical physical conditions regardless of their computer skill.

Research question

Evaluating digital interventions using computer-based surveys can introduce measurement bias, as treatment participants gain greater computer familiarity than control participants during the intervention.

Participants

N = 4,599 human ninth-grade public school students across 138 classrooms in 21 Colombian cities.

Study Design

Three unannounced rounds of paper-and-pencil surveys (baseline, post-course at 1 week, and 6-month follow-up) to minimize familiarization bias and attrition.

Results

Attrition was limited to 13% between baseline and first follow-up, and 10% between baseline and second follow-up, with no differential attrition across treatment and control arms.

Takeaway

Administer offline paper-and-pencil evaluations when testing computer-based educational programs to eliminate modality familiarity bias.

Read alongside the research

Plain-English glossary

Original explanations, informed by the references below. Examples are illustrations, not data from the studies above. The research text stays unchanged.

Follow-up

Collecting more information about participants later in a study to see what has happened to them. The term can also mean the length of time researchers keep track of them.

ExampleResearchers interview the same workers three months after a programme ends. That later interview is a follow-up.

GET-IT glossary · Follow-up ↗Read more about follow-up →

Baseline

The starting measurements in a study. Researchers use these to compare what participants were like at the start with what happens later.

ExampleBefore a training programme starts, researchers measure workers’ earnings. That first measurement is the baseline.

GET-IT glossary · Baseline characteristics ↗Read more about baseline →

Related findings

• Floating Recall Period Data Capture →Methods & evidence

  Instead of asking a participant to report transactions strictly for "Monday through Sunday" when they actually fill out the form on a Tuesday, the app dynamically asks for all activity "since your last survey submission," ensuring every day is covered without gaps or overlaps.

• Active Tracking Attrition Mitigation →Methods & evidence

  Dispatching field teams directly to children's homes when they miss school testing operates like a home delivery service tracking down uncollected mail to ensure every package is accounted for.

• Commercial Ad Platform Algorithmic Randomization Imbalance →Methods & evidence

  Allowing a commercial ad algorithm to handle randomization is like letting a profit-driven tournament host pick matching opponents. The platform optimizes for click rates rather than making sure both starting teams are equally balanced.

• Multi-Source Administrative-Survey Data Triangulation →Methods & evidence

  Think of monitoring a patient's health by combining heart rate monitors and blood tests with direct interviews about how they feel. Combining automated medical readings with verbal feedback provides a complete, accurate picture of health rather than relying on either measurement alone.