Analyzing Model Behavior Under Maximum Likelihood Estimation
A development team is fine-tuning a language model on a dataset of instructions and ideal responses. The goal is for the model to be a direct and concise assistant. For an input like List the primary colors, the ideal response in the training data is Red, yellow, and blue. However, after training, the model consistently generates Of course! The primary colors are red, yellow, and blue.
Assuming the training process is correctly configured to maximize the probability of generating the ideal responses, analyze this situation. Explain why the model might learn to generate the conversational preamble (Of course! The...) despite it not being present in the ideal response for this specific example. Your explanation should focus on how the training objective functions on a token-by-token basis across the entire dataset.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A language model is being fine-tuned on a dataset of instruction-response pairs. Consider the following training example:
- Input:
What is the capital of France? - Correct Response:
Paris
The model processes the input and must predict the first token of the response. Below are two potential probability distributions (States A and B) that the model could generate for this first token at different points during training.
- State A:
{'Paris': 0.15, 'London': 0.10, 'The': 0.08, ...} - State B:
{'Paris': 0.25, 'London': 0.05, 'The': 0.04, ...}
Based on the standard objective for this type of training, which statement provides the most accurate analysis?
- Input:
A language model is being fine-tuned on a dataset. For the input
Translate to French: I love to learn., the correct starting token for the response isJ'. At a particular step in training, the model produces the following probabilities for the first token:Je: 0.35J': 0.25Le: 0.15Mon: 0.10- (all other tokens): 0.15
Given that the training objective is to maximize the likelihood of the correct sequence, how will the training process adjust the model's parameters in the next immediate step for this specific token prediction?
Analyzing Model Behavior Under Maximum Likelihood Estimation