Evaluating Component Independence in a Language Model
A fellow student examines the formula for computing token probabilities, [p_1, ..., p_m] = Softmax_W(Encoder_theta(x)), and claims that the probability distribution for the token at position i, p_i, is calculated based only on the representation of the input token at that same position, x_i. Critically evaluate this claim. Is it correct? Justify your reasoning based on the function of the components in the formula.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Simplified Notation for Parameterized Models
Comparison of Output Probability Meaning: Language Modeling vs. Encoder Pre-training
A language model computes probability distributions for a sequence of tokens
xusing a two-stage process: an encoder with parametersθgenerates representations, which are then passed to a Softmax layer with a weight matrixW. This model is consistently outputting a nearly uniform probability distribution for every token position, meaning every word in the vocabulary is considered almost equally likely, regardless of the input. Which of the following is the most direct and plausible explanation for this behavior?Evaluating Component Independence in a Language Model
A language model calculates the probability distribution for each token in an input sequence,
x, by first generating a sequence of numerical representations and then applying a final transformation. Arrange the following steps in the correct computational order to produce the probability vector,p_i, for the token at a specific positioni.