Learn Before
Troubleshooting a Flawed Reward Model
Based on the fundamental design of a reward model within this type of learning framework, what is the critical error in the engineer's approach, and why does this error lead to the observed problem?
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Prep Sessions
Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Ch.3 Model Alignment and Safety - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Model-Assisted Safety and Rule-Based Reward Models - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
OpenStax Psychology (2nd ed.) Textbook
Related
Notation for the RLHF Reward Model
A system designed to improve language model outputs uses a special component. This component takes a user's initial text (a prompt) and a model-generated response, then outputs a single numerical score. If this component processes two different responses for the exact same prompt, giving 'Response A' a score of 4.1 and 'Response B' a score of -0.5, what is the most accurate interpretation of these scores?
Troubleshooting a Flawed Reward Model
Semantic Completeness in RLHF Reward Models
Within the Reinforcement Learning from Human Feedback (RLHF) framework, the reward model is structured as a neural network that maps an input sequence and an output sequence to a single scalar value.
A reward model in an RLHF framework processes an interaction where a user submits the prompt "Summarize the plot of Hamlet" and receives the completion "Hamlet is a tragedy by William Shakespeare about Prince Hamlet seeking revenge against his uncle Claudius," yielding a result of 2.5. Based on the structure of an RLHF reward model, identify the two token sequence inputs (x and y) and specify what the resulting output value of 2.5 represents.