An alternative to the softmax function for modeling categorical probabilities is the probit model, which assumes that the raw outputs $$\mathbf{o}$$ are corrupted versions of the true labels $$\mathbf{y}$$. This corruption is modeled by adding noise $$\boldsymbol{\epsilon}$$ drawn from a normal distribution. Mathematically, it is expressed as $$\mathbf{y} = \mathbf{o} + \boldsymbol{\epsilon}$$, where $$\epsilon_i \sim \mathcal{N}(0, \sigma^2)$$. While conceptually appealing for explaining variance, the probit model is generally less effective and leads to a more difficult optimization problem compared to softmax regression.

Probit Model

To convert raw, unnormalized outputs $$\mathbf{o}$$ into valid probabilities, the softmax function applies an exponential function to each component and then normalizes them by their sum. The exponentiation ensures that all probabilities are non-negative, while the division ensures that they sum to $$1$$. Mathematically, the predicted probability distribution $$\hat{\mathbf{y}}$$ is defined as:

$$\hat{\mathbf{y}} = \mathrm{softmax}(\mathbf{o}) \quad 	extrm{where}\quad \hat{y}_i = \frac{\exp(o_i)}{\sum_j \exp(o_j)}$$

This guarantees that $$0 \le \hat{y}_i \le 1$$ and $$\sum_j \hat{y}_j = 1$$. Unlike other normalizations or the probit model, the softmax function preserves order and leads to a more well-behaved optimization problem.

Softmax Function

Treating multi-class classification directly as a vector-valued linear regression problem (minimizing the difference between raw outputs $$\mathbf{o}$$ and one-hot labels $$\mathbf{y}$$) is unsatisfactory for probability estimation. First, there is no guarantee that the outputs will sum to $$1$$, which is required for a valid probability distribution. Second, there is no guarantee that the outputs will be non-negative or bounded by $$1$$. These limitations render the solution difficult to interpret probabilistically and brittle to outliers.

Claude

To estimate class probabilities across multiple categories, a classification model requires an output for each class. Using a linear model, this is achieved by defining an affine function for each output. For $$d$$ input features and $$q$$ output categories, the unnormalized outputs (logits) $$\mathbf{o}$$ are computed as $$\mathbf{o} = \mathbf{W}\mathbf{x} + \mathbf{b}$$, where the weight matrix $$\mathbf{W} \in \mathbb{R}^{q 	imes d}$$ contains $$q 	imes d$$ scalars and the bias $$\mathbf{b} \in \mathbb{R}^q$$ contains $$q$$ scalars. Because every output depends on every input feature, this computation represents a fully connected layer in a single-layer neural network.

Learn Before

Related

Learn After