Based on benchmark evaluations on MMLU, how does post-training alignment quantitatively alter GPT-4's Expected Calibration Error (ECE)?
0
1
Tags
Prep Sessions
Frontier Model Dynamics: Scaling Laws, Calibration, and Post-Training Alignment @ University of Michigan - Ann Arbor
Ch.2 Post-Training Analysis and Safety - Frontier Model Dynamics: Scaling Laws, Calibration, and Post-Training Alignment @ University of Michigan - Ann Arbor
Model Calibration and Post-Training Effects - Frontier Model Dynamics: Scaling Laws, Calibration, and Post-Training Alignment @ University of Michigan - Ann Arbor
Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor
Ch.2 Post-Training Alignment and Calibration - Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor
RLHF Effects on Capability and Calibration - Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor
Related
Match each stage, metric value, or procedure to its role in large language model calibration on the MMLU benchmark.
Order the progression of model calibration and confidence behavior from the pre-trained state through post-training alignment.
Explain how the post-training alignment impacted the model's calibration and what the increase in ECE signifies regarding the model's confidence versus its accuracy.
In pre-trained GPT-4, what relationship is observed between the model's predicted probabilities (logprobs) across multiple-choice options and its actual task accuracy on benchmarks like MMLU?
Based on benchmark evaluations on MMLU, how does post-training alignment quantitatively alter GPT-4's Expected Calibration Error (ECE)?