Learn Before
Total Reward as Sum of Segment-Based Scores
The cumulative reward score for an entire output token sequence, represented as , is determined by calculating the sum of the individual reward scores from all its segmented parts. The formal equation for this aggregation is: In this formula, stands for the total number of segments the sequence is divided into, and denotes the computed reward for the -th segment. This total score is typically used to update and train the policy model as usual.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Total Reward as Sum of Segment-Based Scores
Examples of Constant Segment-Based Reward Functions
A team is developing a reward model to score segments of text generated by a language model. The standard approach calculates a segment's score using the initial prompt, the complete generated output, and the specific segment being evaluated. To improve efficiency, a developer suggests modifying the process to calculate the score using only the initial prompt and the specific segment, omitting the rest of the generated output. What is the most significant analytical flaw in this modified approach?
Inputs for Segment-Based Reward Calculation
Role of Context in Segment-Based Reward
Learn After
Application of Segment-Based Total Reward in Policy Training
A language model generates a three-segment response to a user's prompt. A separate reward model evaluates each segment, considering the full context of the prompt and the complete response, and assigns the following scores: Segment 1: 0.8, Segment 2: -0.3, Segment 3: 0.5. According to the principle of aggregating segment-based scores, what is the total reward for the entire generated response?
Analyzing Reward Model Behavior
Calculating a Missing Segment Score