LLM Policy as a Probability Distribution
In the context of reinforcement learning, the policy of a Large Language Model agent is the model's probability distribution over possible outputs. This policy, often denoted by , is equivalent to the conditional probability of generating an output sequence given an input context . This relationship is expressed as .
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.4 Alignment - Foundations of Large Language Models
Related
Fundamental LLM Training Objective
LLM Policy as a Probability Distribution
A language model is given the context: 'The chef carefully added the final, crucial ingredient to the simmering stew: a pinch of...'. The model must predict the next word. Below are the conditional probabilities,
Pr(next_word | context), calculated by two different models for four possible next words.Next Word Model A Probability Model B Probability salt 0.65 0.20 concrete 0.02 0.45 laughter 0.03 0.15 thyme 0.30 0.20 Based on this data, which of the following statements is the most accurate analysis of the models' understanding of the context?
Mathematical Notation for Text Generation Probability
Evaluating Language Model Suitability
Predicting Next-Word Likelihood
Loss Function for Language Modeling
Policy in the Context of LLMs
LLM Policy as a Probability Distribution
Identifying the Agent and Action in a Training Scenario
When a language model is fine-tuned using a system that incorporates human preferences, this process is often conceptualized within a reinforcement learning framework. Which of the following statements correctly analyzes the components of this interaction?
When training a language model using a framework that incorporates human feedback, standard reinforcement learning terminology is used. Match each reinforcement learning term on the left with its corresponding component or concept in this specific language model training context on the right.
Learn After
A research team is training a language model to act as a helpful assistant using methods from reinforcement learning. One researcher is focused on analyzing the model's 'policy' (π) for generating a response given a user's query. Another researcher is analyzing the model's 'conditional probability distribution' (Pr) over all possible responses for the same query. What is the relationship between the 'policy' and the 'conditional probability distribution' in this context?
Modifying a Chatbot's Behavior
When applying reinforcement learning to a language model, the model's policy, denoted as π(y|x), is a separate computational function that is trained to approximate the model's core conditional probability distribution, Pr(y|x).
Policy Notation for Autoregressive Models ()