Loss Function for Conditional Probability Distributions (Loss(\\text{Pr}^t(\\cdot|\\cdot), \\text{Pr}_\\theta^s(\\cdot|\\cdot), \\mathbf{x}))
The general loss function, denoted as Loss(\\text{Pr}^t(\\cdot|\\cdot), \\text{Pr}_\\theta^s(\\cdot|\\cdot), \\mathbf{x}), measures the discrepancy between a target conditional probability distribution, , and a student model's predicted distribution, \\text{Pr}_\\theta^s(\\cdot|\\cdot), for a given input . The training objective is to adjust the parameters to minimize this loss, aligning the student's output distribution with the target distribution.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
A machine learning team is developing a compact, efficient language model, which we'll call model 's'. The model's behavior is governed by a set of tunable weights, denoted by θ. For a given task, the model receives a simplified context input, c', and a latent variable, z, and then generates a probability distribution over all possible outputs. Which of the following expressions correctly represents this model's output probability distribution?
In the expression , which describes a model's output probability distribution, match each symbol to its correct description.
Applying the Student Model Probability Notation
General Loss Function for Knowledge Distillation
Loss Function for Conditional Probability Distributions (Loss(\\text{Pr}^t(\\cdot|\\cdot), \\text{Pr}_\\theta^s(\\cdot|\\cdot), \\mathbf{x}))
KL Divergence Loss for Knowledge Distillation
Cross-Entropy Loss for Knowledge Distillation
A large, complex language model is used to generate target probabilities for training a smaller, more efficient model. For the input sentence 'The cat sat on the ___', the large model could produce different probability distributions for the next word. Which of the following distributions, representing , would provide the most informative and nuanced training signal for the smaller model?
Value of the Teacher's Probability Distribution
In a knowledge distillation process for a machine translation task, a large 'teacher' model translates the sentence 'Je suis content' from French to English. Instead of just outputting 'I am happy', the teacher model produces a full probability distribution over the entire English vocabulary for the next words. Which statement best analyzes the significance of this probability distribution () for training the smaller 'student' model?
General Loss Function for Knowledge Distillation
Loss Function for Conditional Probability Distributions (Loss(\\text{Pr}^t(\\cdot|\\cdot), \\text{Pr}_\\theta^s(\\cdot|\\cdot), \\mathbf{x}))
Learn After
A language model is being trained to predict the next word in a sentence. For the input context 'The sun is shining...', the ideal (target) probability distribution, denoted as , gives a high probability to the word 'brightly'. The model's performance is measured by a loss function that compares the model's predicted probability distribution, , to the target distribution.
Consider two different sets of model parameters, θ₁ and θ₂:
- With parameters θ₁, the model's distribution predicts 'brightly' with a high probability.
- With parameters θ₂, the model's distribution predicts 'darkly' with a high probability.
Which of the following statements correctly analyzes the relationship between the parameters and the loss function for this specific input?
Interpreting a Model's Training Step
Comparing Model Performance via Loss