Probability Distribution Formula for an Encoder-Softmax Language Model

Output Probability Calculation in Transformer Language Models

Next-Token Probability Calculation in Autoregressive Decoders

A neural network produces a final matrix of hidden state vectors, H, with dimensions [sequence_length × hidden_dimension]. To generate a probability distribution over a vocabulary of size V for each position in the sequence, a parameterized Softmax layer is used, which computes Softmax(H ⋅ W). What is the primary role and required shape of the weight matrix W in this operation?

Debugging a Parameterized Softmax Layer

A parameterized Softmax layer is used to convert a sequence of hidden state vectors into a sequence of probability distributions over a vocabulary. Arrange the following steps of this process into the correct chronological order.

Parameterized Softmax Function Notation