Model Parameter Selection via Likelihood
A language model is being trained on a small dataset consisting of two sequences. You are evaluating two different sets of model parameters, and . The model's calculated log-probabilities for each sequence under each parameter set are shown in the table below. Based on the principle of selecting parameters that maximize the total log-probability across the entire dataset, which parameter set should be chosen? Justify your answer by showing the calculation for each set.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.2 Generative Models - Foundations of Large Language Models
Ch.4 Alignment - Foundations of Large Language Models
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Maximum Likelihood Estimation for Sequential Data
Fine-Tuning as Maximum Likelihood Estimation
Log-Probability Decomposition for Efficient Multi-Turn Dialogue Training
A language model is being trained on a dataset containing a mix of very short sequences and a few extremely long sequences. A developer observes that the overall training objective, which is the sum of the log-probabilities of all sequences in the dataset, seems to be disproportionately influenced by the model's performance on the few long sequences. Which of the following best explains this observation?
Model Parameter Selection via Likelihood
A language model is being trained on a large dataset of text sequences. After a single parameter update, the model's calculated log-probability for one specific sequence in the dataset increases by 2.5, while the log-probabilities for all other sequences in the dataset remain exactly the same. How does this change affect the overall maximum likelihood training objective for the entire dataset?
Standard Optimization Objective for Transformer Language Models