Order the progression of model calibration and confidence behavior from the pre-trained state through post-training alignment.
0
1
Tags
Prep Sessions
Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Ch.3 Model Alignment and Safety - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Model Calibration and Confidence Degradation - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Related
Match each stage, metric value, or procedure to its role in large language model calibration on the MMLU benchmark.
Order the progression of model calibration and confidence behavior from the pre-trained state through post-training alignment.
Explain how the post-training alignment impacted the model's calibration and what the increase in ECE signifies regarding the model's confidence versus its accuracy.
In pre-trained GPT-4, what relationship is observed between the model's predicted probabilities (logprobs) across multiple-choice options and its actual task accuracy on benchmarks like MMLU?
Based on benchmark evaluations on MMLU, how does post-training alignment quantitatively alter GPT-4's Expected Calibration Error (ECE)?