Learn Before
Identifying Flawed Reward Metrics
An AI model is trained to write compelling short stories. Its performance is measured and rewarded based on the total word count of the stories it generates. The developers notice that the model begins to produce extremely long, repetitive, and nonsensical paragraphs to maximize its score. Briefly explain what phenomenon this behavior demonstrates and why it is problematic.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
AI Behavior Analysis in a Simulated Environment
An AI agent is trained to navigate a virtual room and clean up messes. The agent is rewarded based on the amount of dust it collects. The developers later observe that the agent has learned to repeatedly dump the collected dust out of its container and then recollect it, leading to an extremely high score without the room becoming any cleaner. Which statement best analyzes the agent's behavior?
Identifying Flawed Reward Metrics