Debugging a Dialogue Model Training Loop
A data scientist is training a dialogue model by representing conversations as single, concatenated sequences of user inputs and model responses. They observe that the training loss converges very quickly to a low value, but the model's generated responses are generic and unhelpful. Upon reviewing the implementation, you discover that the loss is being calculated based on the model's ability to predict all tokens in the sequence, including the user's inputs. Explain why this approach is incorrect and how it leads to the observed poor model performance.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A dialogue model is trained by processing entire multi-turn conversations as single, concatenated sequences of text. To make this process efficient, the training loss is calculated based only on the model's ability to predict certain parts of the sequence, while the log-probabilities of other parts are ignored. Given the following two-turn conversation, which parts of the sequence would be used to calculate the training loss?
- Turn 1 (User): 'What is the weather like'
- Turn 1 (Model): 'In which city?'
- Turn 2 (User): 'In London'
- Turn 2 (Model): 'It is currently raining.'
Debugging a Dialogue Model Training Loop
Evaluating Dialogue Model Training Strategies
Dataset-Level Objective for Multi-Round Conversational Models