Learn Before
Parameterized Softmax Function Notation
The notation represents the softmax function parameterized by a set of weights, denoted by the subscript . This signifies that the function's output is dependent on these weights, which are typically learned during the training process of a machine learning model.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Probability Distribution Formula for an Encoder-Softmax Language Model
Output Probability Calculation in Transformer Language Models
Next-Token Probability Calculation in Autoregressive Decoders
A neural network produces a final matrix of hidden state vectors, H, with dimensions [sequence_length × hidden_dimension]. To generate a probability distribution over a vocabulary of size V for each position in the sequence, a parameterized Softmax layer is used, which computes Softmax(H ⋅ W). What is the primary role and required shape of the weight matrix W in this operation?
Debugging a Parameterized Softmax Layer
A parameterized Softmax layer is used to convert a sequence of hidden state vectors into a sequence of probability distributions over a vocabulary. Arrange the following steps of this process into the correct chronological order.
Parameterized Softmax Function Notation
Learn After
Two different machine learning models, Model A and Model B, use a parameterized function to convert a vector of raw scores into a probability distribution. Model A uses the function denoted as , and Model B uses . When given the exact same input vector, Model A produces the output
[0.7, 0.2, 0.1]and Model B produces[0.3, 0.6, 0.1]. What is the most logical conclusion that can be drawn from this observation?Interpreting Function Notation
Consider two distinct machine learning models that both utilize a function denoted as . If both models are configured with the exact same weight vector , they are guaranteed to produce identical output probability distributions when given the same input vector.