Tracking objective factual recall across multiple panel waves isolates learning from subjective opinion shifts.

Download image · 1200 × 630 PNG

Research procedure

Repeat the same factual knowledge test

  1. Baseline before briefing materials
  2. Midterm after first deliberative session
  3. Final test at weekend completion

Illustrative summary of the source’s eight-item multiple-choice battery. The same questions cover topic-specific and regional facts at baseline, midterm and final evaluation. No knowledge gains are added.

See source →

How

When to use

Public consultations often face skepticism regarding whether citizen discussion produces genuine learning or merely superficial opinion shifts. Evaluators use this method to quantify objective knowledge gains among deliberative participants across local policy topics. It enables program managers to demonstrate measurable increases in policy literacy following information provision and debate.

Administer a structured battery of factual, multiple-choice questions across baseline, midterm, and final evaluation waves to measure citizen information acquisition.

  1. Draft an eight-item factual question battery covering topic-specific knowledge items and general regional political facts.
  2. Formulate specific multiple-choice questions with objective, verifiable answers regarding local demographics, economic trends, statutory regulations, and facility operations.
  3. Administer the baseline knowledge battery during initial participant recruitment interviews prior to mailing briefing materials.
  4. Readminister the identical knowledge battery on-site at midterm after participants complete their first deliberative session.
  5. Administer the final knowledge battery at the conclusion of the deliberative weekend to track total information gains.

Why it works

Repeated testing reveals how briefing materials and structured group discussions directly enhance citizen political sophistication.

Example

Evaluators tested 132 deliberative poll participants in New Haven, Connecticut, on eight factual items covering airport operations, tax sharing, and regional growth. Overall participant knowledge increased by 22.3 percentage points from baseline to final testing, with the largest knowledge gain occurring on revenue-sharing items, which rose from 25.4% to 59.9% correct.

About the study

Participants

Initial interview sample of 1,032 residents from 15 towns surrounding New Haven, Connecticut, with 133 attending the event and 132 completing questionnaires. Participants were 50.8% male, 49.2% female, 72.7% Caucasian, 12.1% African American, 3% Hispanic or Latino/a, and 90.2% registered voters, with an average age of 50.1 years.

Study Design

Longitudinal panel survey within a split-half randomized field experiment administering an eight-item factual knowledge battery across three waves: baseline telephone interview (T1), midterm post-session questionnaire (T2), and event completion questionnaire (T3).

Results

Overall factual knowledge across all eight items increased by 22.3%, rising from 36.8% correct at T1 to 55.3% at T2 and 59.1% at T3. Knowledge on revenue-sharing items increased by 34.5%, rising from 25.4% at T1 to 49.2% at T2 and 59.9% at T3. General regional knowledge increased by 19.5%, rising from 41.7% at T1 to 60.6% at T2 and 61.2% at T3. Airport knowledge increased by 15.5%, rising from 38.6% at T1 to 50.8% at T2 and 54.2% at T3.

Source

See source →