University of Michigan - Ann Arbor

The model-assisted safety pipeline is an alignment framework that uses the language models themselves as tools to steer model behavior toward safe and nuanced responses. Designed to address failure modes in standard RLHF—such as generating unsafe advice or becoming overly cautious by hedging and refusing benign queries—the framework consists of two primary components: an expanded set of safety-relevant training prompts and rule-based reward models (RBRMs) that supply targeted feedback during reinforcement learning fine-tuning.

Model-Assisted Safety Pipeline

Which set of failure modes in standard reinforcement learning from human feedback (RLHF) is the model-assisted safety pipeline designed to address?

Within the model-assisted safety pipeline, what specific role do rule-based reward models (RBRMs) perform during reinforcement learning fine-tuning?

Rule-based reward models (RBRMs) are zero-shot language model classifiers that supply additional reward signals to the policy model during reinforcement learning fine-tuning. Rather than relying solely on scalar human preference models, RBRMs provide granular, rule-governed feedback to enforce safety policies, specifically rewarding the model for correctly refusing harmful requests while preventing unnecessary refusals on safe prompts.

Learn Before

Related