Safety mitigations and post-training interventions substantially improve GPT-4's safety performance relative to GPT-3.5. Specifically, GPT-4 exhibits an $82\%$ decrease in the tendency to respond to requests for disallowed content. On sensitive requests, such as those regarding medical advice or self-harm, GPT-4 complies with policy guidelines $29\%$ more often. Additionally, on the RealToxicityPrompts benchmark, GPT-4 produces toxic generations only $0.73\%$ of the time, compared to $6.48\%$ for GPT-3.5.

Safety Metric Improvements in GPT-4

The model-assisted safety pipeline is an alignment framework that uses the language models themselves as tools to steer model behavior toward safe and nuanced responses. Designed to address failure modes in standard RLHF—such as generating unsafe advice or becoming overly cautious by hedging and refusing benign queries—the framework consists of two primary components: an expanded set of safety-relevant training prompts and rule-based reward models (RBRMs) that supply targeted feedback during reinforcement learning fine-tuning.

Model-Assisted Safety Pipeline

A rule-based reward model (RBRM) processes three inputs: an optional prompt, the response output generated by the policy model, and a human-written rubric structured with explicit evaluation rules (often in multiple-choice format). The classifier evaluates the response against the rubric categories—such as a refusal in the desired style, a refusal in an undesired style (e.g., rambling or evasive), disallowed content, or a safe non-refusal—allowing the system to assign tailored rewards or penalties based on the classification outcome.

Inputs and Rubric Classification Mechanism of RBRMs

University of Michigan - Ann Arbor

Ch.2 Post-Training Analysis and Safety - Frontier Model Dynamics: Scaling Laws, Calibration, and Post-Training Alignment @ University of Michigan - Ann Arbor

Model Calibration and Post-Training Effects - Frontier Model Dynamics: Scaling Laws, Calibration, and Post-Training Alignment @ University of Michigan - Ann Arbor

Safety Metrics and Refusal Behavior - Frontier Model Dynamics: Scaling Laws, Calibration, and Post-Training Alignment @ University of Michigan - Ann Arbor

Learn Before

Related

Learn After