No More Grading: Incentive Compatible Peer Assessment in a Project-Based Course
Grading team projects is hard; getting students to evaluate each other honestly is harder. We describe a peer-assessment system refined over eight years that uses SVD on residualized evaluations to extract a robust quality signal and rewards careful, honest assessment. Download Paper Abstract We describe a system for grading collaborative student work that has been developed and refined over eight years of a project-based undergraduate course. Students complete four team projects per semester, with teams reassigned each round. After each project, every student evaluates other teams’ deliverables, assesses their own teammates’ contributions, and predicts the scores they themselves will receive. These three streams of assessment data are combined using singular value decomposition (SVD) applied to residualized evaluation matrices. Residualization removes evaluator-specific biases (the tendency to rate generously or harshly); SVD then extracts the dominant latent quality axis from the bias-corrected data, weighting questions by their informativeness. Scores are aggregated via medians for robustness to outlier evaluators. Each student’s composite grade reflects five components: team quality, individual contribution as assessed by teammates, evaluator discrimination (rewarding careful assessment of others), and two measures of self-assessment accuracy (penalizing the gap between predicted and actual peer ratings). We map composite scores to letter grades using an anchor-point method that keys grade boundaries to the median performance of the top five students, spacing bands at one-third standard deviation intervals. The system also feeds assessment information forward into team composition. Teams are formed via a stratified round-robin algorithm that spreads technical skill evenly across groups; for subsequent projects, self-reported skill is replaced by peer-assessed skill. A conflict-avoidance mechanism uses “would you work with this person again” responses to keep incompatible students apart. We present the mathematical foundations in enough detail for replication, discuss incentive properties (robustness to strategic manipulation, encouragement of honest and thoughtful evaluation), and illustrate the procedure with a worked numerical example. ...