University of Michigan - Ann Arbor

Rule-based reward models (RBRMs) are zero-shot language model classifiers that supply additional reward signals to the policy model during reinforcement learning fine-tuning. Rather than relying solely on scalar human preference models, RBRMs provide granular, rule-governed feedback to enforce safety policies, specifically rewarding the model for correctly refusing harmful requests while preventing unnecessary refusals on safe prompts.

Rule-Based Reward Models (RBRMs)

How are rule-based reward models (RBRMs) categorized in terms of classifier type?

Rule-based reward models (RBRMs) rely solely on scalar human preference models to enforce safety policies.

During which training phase do rule-based reward models (RBRMs) supply additional reward signals to the policy model?

Explain the two key safety objectives that rule-based reward models (RBRMs) are designed to balance regarding model refusals.

A rule-based reward model (RBRM) processes three inputs: an optional prompt, the response output generated by the policy model, and a human-written rubric structured with explicit evaluation rules (often in multiple-choice format). The classifier evaluates the response against the rubric categories—such as a refusal in the desired style, a refusal in an undesired style (e.g., rambling or evasive), disallowed content, or a safe non-refusal—allowing the system to assign tailored rewards or penalties based on the classification outcome.

Learn Before

Related