University of Michigan - Ann Arbor

The output layer of softmax regression generates a probability distribution $$\hat{\mathbf{y}}$$, where each entry $$\hat{y}_i$$ represents the predicted probability that the input belongs to a particular class. These probabilities are calculated by applying the softmax activation function to the raw output scores $$\mathbf{o}$$. To ensure the outputs represent valid probabilities, the operation exponentiates each score and normalizes it by the sum of all exponentiated scores:

$$\hat{y}_i = \frac{\exp(o_i)}{\sum_j \exp(o_j)}$$

This transforms the unconstrained linear model outputs into a valid probability distribution.

Output Layer of Softmax Regression

Say we have $z^{[L]} = \begin{bmatrix} 2 \\ -1 \\ 3 \\ 7 \end{bmatrix}$.

Then, $t = \begin{bmatrix} e^2 \\ e^{-1} \\ e^3 \\ e^7 \end{bmatrix} = \begin{bmatrix} 7.39 \\ 0.37 \\ 20.1 \\ 1096.6 \end{bmatrix}$.

We sum the entries in $t$ to yield the denominator of the activation: $7.39 + 0.37 + 20.1 + 1096.6 = 1124.5$.

Finally, normalize the probabilities by dividing each entry in $t$ by the summation we just computed: $a^{[L]} = \begin{bmatrix} 7.39/1124.5 \\ 0.37/1124.5 \\ 20.1/1124.5 \\ 1096.6/1124.5 \end{bmatrix} = \begin{bmatrix} 0.007 \\ 0.0003 \\ 0.02 \\ 97.5 \end{bmatrix}$

Sample Calculation of Softmax Output Layer

Example of a SoftMax activation transformation

In probabilistic classification, a model's output is a probability distribution over a set of possible labels. To determine the final prediction, a decision rule is applied which selects the single label that corresponds to the highest probability within this distribution.

Maximum Probability Decision Rule

Consider a 4-class classification problem where the final layer of a model produces the following pre-activation scores for a single input: `[1.0, 2.0, 1.5, 5.0]`. The model then uses an activation function that exponentiates each score and normalizes the results to produce a probability distribution. Without performing the full calculation, which of the following statements best describes the resulting probability distribution?

A model's final layer produces a vector of scores, `[2.0, 1.0, 0.1]`, for a 3-class classification problem. To convert these scores into probabilities, each score is first exponentiated. Then, each exponentiated score is divided by the sum of all the exponentiated scores. What is the resulting probability for the first class (corresponding to the score 2.0)? Provide your answer as a decimal rounded to two places.

Calculating an Output Probability

An engineer is building a model to classify text into one of three sentiment categories: 'Positive', 'Negative', or 'Neutral'. For a specific piece of text, the layer just before the final activation function produces the numerical scores `[3.5, -2.0, 3.5]` for 'Positive', 'Negative', and 'Neutral' respectively. The engineer observes that the final output probabilities for 'Positive' and 'Neutral' are identical and significantly higher than the probability for 'Negative'. Evaluate this observation. Is this the expected behavior? Justify your reasoning by describing how the final activation function processes its input scores to produce a probability distribution.

Classifier Output Analysis

For any fully connected neural network layer with $$d$$ inputs and $$q$$ outputs, the parametrization and computational cost is quadratic, specifically $$\mathcal{O}(dq)$$. This high cost can become prohibitive in practical deep learning applications.

Learn Before

Related