Subordinates observe daily peer interactions and know whether a manager's performance ratings reflect actual workplace effort.

Download image · 1200 × 630 PNG

Research procedure

Measure perceived manager rating error

  1. Ask teachers about ratings of three peers
  2. Code and average perceived rating errors by manager
  3. Standardize the manager inaccuracy index

Illustrative summary of the source-described peer appraisal survey using a five-step appraisal scale. Teachers report whether ratings are accurate or off by less than, about or more than one category.

See source →

How

When to use

Organizations struggle to measure whether managers accurately evaluate employee effort under subjective incentive contracts. Unobserved manager rating error undermines employee motivation and creates contract noise. Gathering anonymous peer assessments of manager accuracy allows human resource teams to diagnose evaluation inaccuracy and isolate its impact on employee performance.

Measure manager appraisal accuracy by surveying employees about the rating error of their direct supervisors across three peer colleagues.

  1. Select three peer colleagues in the same organization who work under the same manager for each surveyed employee.
  2. Ask employees on endline surveys to evaluate how accurately their manager rates the performance of those three peer colleagues on a five-step appraisal scale.
  3. Code evaluation categories as accurate, off by less than one category, off by about one category, or off by more than one category.
  4. Calculate an average perceived inaccuracy score for each manager across all peer evaluations submitted by their subordinates.
  5. Standardize manager inaccuracy scores into z-scores to construct an instrumental variable for subjective contract noise.

Why it works

Aggregating peer assessments across multiple colleagues creates a reliable measure of manager inaccuracy that predicts perceived contract noise without relying on manager self-reports.

Example

Researchers surveyed teachers across private schools in Pakistan to assess whether school managers accurately rated three fellow teachers on a 5-step appraisal scale. The resulting peer-based inaccuracy index revealed that a 1 standard deviation increase in manager inaccuracy reduced subjective incentive effectiveness on student test scores by 0.211 standard deviations.

About the study

Participants

6,080 teachers and 189 managers across 255 school clusters in Pakistan (440,595 student test observations).

Study Design

Randomized controlled trial (RCT) using a peer-appraisal survey instrument interacted with subjective raise contracts to construct a two-stage least squares instrument for perceived contract noise.

Results

Teachers reported that managers mis-rated peer performance by an average of 0.8 appraisal categories, while managers in the most inaccurate quintile mis-rated peers by more than 2 categories; a 1 standard deviation increase in manager rating inaccuracy interacted with subjective treatment reduced student test scores by 0.211 standard deviations (p < 0.05).

Source

See source →