Learn Before
Impact of Reward Model Flaws on Value Function Estimation
An agent is being trained to navigate a maze. Its reward model is designed to give a small positive signal for each step taken that does not hit a wall, and a large positive signal for reaching the exit. However, due to a flaw, the model also provides a moderately high positive signal for moving into a specific dead-end corridor. Analyze the likely effect of this flaw on the agent's computed long-term value for states within and near this corridor. How might this flawed value estimation, in turn, influence the agent's final learned path through the maze?
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Impact of Reward Model Flaws on Value Function Estimation
A reinforcement learning agent is trained to find the exit in a maze. Two reward models are proposed. Model A gives a reward of +100 for reaching the exit and 0 for every other step. Model B gives +100 for reaching the exit but also a -1 penalty for each step taken. How will the value function derived from Model B most likely differ from the one derived from Model A for states that are not the exit?
Diagnosing Undesirable Agent Behavior
In a reinforcement learning framework, it is possible to compute a meaningful long-term value function for a policy even if the reward model consistently provides random, uninformative feedback for every action.