To maximize computational efficiency, the forward pass of a softmax regression model is typically vectorized across minibatches. For a minibatch of inputs $$\mathbf{X} \in \mathbb{R}^{n 	imes d}$$ containing $$n$$ examples with $$d$$ features, and parameters $$\mathbf{W} \in \mathbb{R}^{d 	imes q}$$ (weights) and $$\mathbf{b} \in \mathbb{R}^{1 	imes q}$$ (biases), the unnormalized logits are computed using the affine transformation $$\mathbf{O} = \mathbf{X} \mathbf{W} + \mathbf{b}$$. The softmax function is then applied rowwise to $$\mathbf{O}$$ to yield the normalized class probabilities $$\hat{\mathbf{Y}} = \mathrm{softmax}(\mathbf{O})$$ for the entire batch simultaneously.

Vectorized Minibatch Softmax Regression

The softmax function maps a matrix of raw scalar outputs, denoted as $$\mathbf{X}$$, into a valid probability distribution. It transforms each element into a non-negative number and ensures that each row sums to $$1$$. Computing the softmax involves three steps: first, exponentiating each term; second, computing a normalization constant by summing over each row; and third, dividing each row by its corresponding normalization constant. The mathematical formula is expressed as:

$$\mathrm{softmax}(\mathbf{X})_{ij} = \frac{\exp(\mathbf{X}_{ij})}{\sum_k \exp(\mathbf{X}_{ik})}$$

Claude

Because a non-reduction sum retains its original axes after summing along a specific dimension, the resulting tensor can be directly used in elementwise operations with the original tensor via broadcasting. For instance, dividing a matrix by its non-reduction row sum seamlessly triggers the broadcasting mechanism to expand the sum tensor, thereby generating a new matrix where the elements of each row mathematically sum up to $$1$$.

Learn Before

Related

Learn After