Google

When applying reinforcement learning to Large Language Models, an action, denoted as $$a$$, corresponds to a possible decision the agent can make. Specifically, an action represents a predicted token chosen from the model's vocabulary.

Action in the Context of LLMs

The policy for a Large Language Model is formally expressed as a conditional probability. Specifically, the policy of taking an action $$a$$ in a state $$s$$, denoted as $$\pi(a|s)$$, is the probability of generating the next token $$y_t$$ given the input prompt $$\mathbf{x}$$ and the sequence of previously generated tokens $$\mathbf{y}_{< t}$$. The formula is: $$\pi(a|s) = \Pr(y_t | \mathbf{x}, \mathbf{y}_{< t})$$. In this formulation, the action $$a$$ corresponds to the predicted token $$y_t$$, and the state $$s$$ corresponds to the context sequence $$(\mathbf{x}, \mathbf{y}_{< t})$$.

Policy Formula for LLMs in Reinforcement Learning

A language model is generating a response to the prompt 'The best way to learn a new skill is to...'. So far, it has produced the sequence 'The best way to learn a new skill is to practice'. At this exact point in the generation process, what constitutes the model's next 'action' within a reinforcement learning framework?

Consider two reinforcement learning agents: one playing a chess game and another generating text. Contrast the nature of an 'action' for the chess-playing agent versus the text-generating agent. Specifically, what does a single action entail in each scenario?

Comparing 'Action' in Different Reinforcement Learning Scenarios

Given the following scenario, precisely describe the 'action' the language model has just performed within its reinforcement learning framework.

Learn Before

Related