An alternative to the softmax function for modeling categorical probabilities is the probit model, which assumes that the raw outputs $$\mathbf{o}$$ are corrupted versions of the true labels $$\mathbf{y}$$. This corruption is modeled by adding noise $$\boldsymbol{\epsilon}$$ drawn from a normal distribution. Mathematically, it is expressed as $$\mathbf{y} = \mathbf{o} + \boldsymbol{\epsilon}$$, where $$\epsilon_i \sim \mathcal{N}(0, \sigma^2)$$. While conceptually appealing for explaining variance, the probit model is generally less effective and leads to a more difficult optimization problem compared to softmax regression.

Claude

Treating multi-class classification directly as a vector-valued linear regression problem (minimizing the difference between raw outputs $$\mathbf{o}$$ and one-hot labels $$\mathbf{y}$$) is unsatisfactory for probability estimation. First, there is no guarantee that the outputs will sum to $$1$$, which is required for a valid probability distribution. Second, there is no guarantee that the outputs will be non-negative or bounded by $$1$$. These limitations render the solution difficult to interpret probabilistically and brittle to outliers.

Limitations of Direct Linear Regression for Classification

Dive into Deep Learning

Probit Model

To convert raw, unnormalized outputs $$\mathbf{o}$$ into valid probabilities, the softmax function applies an exponential function to each component and then normalizes them by their sum. The exponentiation ensures that all probabilities are non-negative, while the division ensures that they sum to $$1$$. Mathematically, the predicted probability distribution $$\hat{\mathbf{y}}$$ is defined as:

$$\hat{\mathbf{y}} = \mathrm{softmax}(\mathbf{o}) \quad 	extrm{where}\quad \hat{y}_i = \frac{\exp(o_i)}{\sum_j \exp(o_j)}$$

This guarantees that $$0 \le \hat{y}_i \le 1$$ and $$\sum_j \hat{y}_j = 1$$. Unlike other normalizations or the probit model, the softmax function preserves order and leads to a more well-behaved optimization problem.

Learn Before

Related