Degradation of Model Calibration from Post-Training Alignment
Post-training alignment (such as PPO reinforcement learning) can significantly degrade the calibration of large language models. In pre-trained models such as GPT-4, predicted probabilities (logprobs) across multiple-choice options track actual task accuracy closely, yielding near-perfect calibration with an Expected Calibration Error (ECE) of $0.007on benchmarks like MMLU. However, subsequent post-training alignment impairs this correspondence, substantially inflating calibration error to an ECE of $0.074 and causing the model's confidence to diverge from its true probability of being correct.
0
1
Tags
Prep Sessions
Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Ch.3 Model Alignment and Safety - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Model Calibration and Confidence Degradation - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Learn After
Match each stage, metric value, or procedure to its role in large language model calibration on the MMLU benchmark.
Order the progression of model calibration and confidence behavior from the pre-trained state through post-training alignment.
Explain how the post-training alignment impacted the model's calibration and what the increase in ECE signifies regarding the model's confidence versus its accuracy.