Diagnosing Training Instability in a Language Model
Given the following description of a language model's training process, identify the most likely immediate cause of the issue and propose a standard adjustment to the optimization algorithm that could resolve it. Justify your reasoning.
0
1
Tags
Data Science
Foundations of Large Language Models Course
Computing Sciences
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Efficient Attention Models
An engineer is training a neural network for a next-word prediction task. During each training iteration, the model is provided with the correct preceding words from the training data to predict the next word at each position in a sequence. The model is designed to calculate the prediction errors for all positions in the sequence simultaneously within a single computational pass. Which of the following best explains the architectural property that is essential for this parallel and efficient training approach?
Diagnosing Training Instability in a Language Model
A team is training a large neural network for a text generation task. The training process involves iteratively adjusting the network's internal parameters to maximize the likelihood of the text in a large dataset. Arrange the following core steps of a single training iteration into the correct chronological order.