When training a multi-round dialogue model, an engineer chooses to concatenate an entire K-turn conversation into a single sequence and calculate the loss for all K responses in one forward pass. What is the primary advantage of this approach compared to performing K separate forward passes, where each pass processes an incrementally longer conversational history?
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Concatenated Sequence Representation for Multi-Turn Dialogue
Optimizing Dialogue Model Training Efficiency
When training a multi-round dialogue model, an engineer chooses to concatenate an entire K-turn conversation into a single sequence and calculate the loss for all K responses in one forward pass. What is the primary advantage of this approach compared to performing K separate forward passes, where each pass processes an incrementally longer conversational history?
When training a multi-round dialogue model by concatenating the entire conversation into a single sequence for one forward pass, the calculation of the loss for the first model response is influenced by the content of the final user input in the dialogue.