University of Michigan - Ann Arbor

Post-training alignment (such as PPO reinforcement learning) can significantly degrade the calibration of large language models. In pre-trained models such as GPT-4, predicted probabilities (logprobs) across multiple-choice options track actual task accuracy closely, yielding near-perfect calibration with an Expected Calibration Error (ECE) of $0.007$ on benchmarks like MMLU. However, subsequent post-training alignment impairs this correspondence, substantially inflating calibration error to an ECE of $0.074$ and causing the model's confidence to diverge from its true probability of being correct.

Degradation of Model Calibration from Post-Training Alignment

Match each stage, metric value, or procedure to its role in large language model calibration on the MMLU benchmark.

Order the progression of model calibration and confidence behavior from the pre-trained state through post-training alignment.

Explain how the post-training alignment impacted the model's calibration and what the increase in ECE signifies regarding the model's confidence versus its accuracy.

Learn Before

Related