Learn Before
Diagnosing Issues in LLM Reinforcement Learning
Based on the scenario described in the case study, which single component of the standard Proximal Policy Optimization (PPO) objective is most likely misconfigured or has its coefficient set too low? Explain how this component is designed to prevent both of the observed problems.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Parameter Update at the Reference Policy Point in PPO
PPO Objective Formula for LLM Training in RLHF
Diagnosing Issues in LLM Reinforcement Learning
In the context of fine-tuning a language model with reinforcement learning, the optimization objective often includes a penalty term that measures the divergence from an initial reference policy. What is the most critical trade-off this penalty term is designed to manage?
In the context of fine-tuning a language model with reinforcement learning, the optimization objective is composed of several key elements. Match each element with its primary function in the training process.