Learn Before
Impact of a Biased Reward Model on Value Function Training
An AI agent is being trained to generate helpful summaries. Its state-value function, which estimates the expected future rewards for a given summary-in-progress, is trained using a separate reward model. Imagine this reward model has a flaw: it is heavily biased towards summaries that include specific, uncommon keywords, regardless of the summary's overall quality or relevance. Describe two specific consequences this bias will have on the trained state-value function and the agent's resulting summary-generation behavior.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Value Function Loss in RLHF
An AI system is being trained to generate helpful multi-turn dialogues. A state-value function, which estimates the total future reward from the current point in the conversation, is updated using rewards from a separate reward model. The development team observes that the value function consistently assigns very low values to all conversational turns except the very last one, even when the intermediate turns are crucial for a successful outcome. This causes the AI to prematurely end conversations. Which of the following is the most likely cause of this specific issue?
Impact of a Biased Reward Model on Value Function Training
Advantage Function as TD Error in RLHF
Arrange the following events in the correct chronological order to describe a single update step for a value function that relies on a separate reward model.