Evaluating Model Parameters via Distribution Matching
A development team is fine-tuning a small language model. Their training objective is to make the small model's output probability distribution for the next word as close as possible to the output distribution of a larger, high-performing 'teacher' model. For a given input, the team is evaluating two different sets of parameters for their small model. Based on the data below, which parameter set is better according to their objective, and why?
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Objective Function for Student Model Training via Knowledge Distillation
A team is training a compact 'student' model to emulate a powerful 'teacher' model. The training objective is to minimize a loss function that measures the divergence between the probability distributions of the student model's outputs and the teacher model's outputs for a given set of inputs. What is the primary goal of this optimization process?
Evaluating Model Parameters via Distribution Matching
Consider an optimization process where a model's parameters are adjusted to minimize a loss function that measures the difference between the model's output distribution and a target distribution over a dataset
D'. True or False: Increasing the size and diversity of the datasetD'will always guarantee a better match to the target distribution, resulting in a lower final loss value.