Post-training alignment (such as PPO reinforcement learning) can significantly degrade the calibration of large language models. In pre-trained models such as GPT-4, predicted probabilities (logprobs) across multiple-choice options track actual task accuracy closely, yielding near-perfect calibration with an Expected Calibration Error (ECE) of $0.007$ on benchmarks like MMLU. However, subsequent post-training alignment impairs this correspondence, substantially inflating calibration error to an ECE of $0.074$ and causing the model's confidence to diverge from its true probability of being correct.

Degradation of Model Calibration from Post-Training Alignment

University of Michigan - Ann Arbor

Ch.2 Post-Training Analysis and Safety - Frontier Model Dynamics: Scaling Laws, Calibration, and Post-Training Alignment @ University of Michigan - Ann Arbor

Safety Metrics and Refusal Behavior - Frontier Model Dynamics: Scaling Laws, Calibration, and Post-Training Alignment @ University of Michigan - Ann Arbor

Model Calibration and Post-Training Effects - Frontier Model Dynamics: Scaling Laws, Calibration, and Post-Training Alignment @ University of Michigan - Ann Arbor

Learn Before

Related

Learn After