Because a neural network's activation function $$\phi$$ applies elementwise, computing the gradient of the objective function $$J$$ with respect to the pre-activation intermediate variable $$\mathbf{z} \in \mathbb{R}^h$$ requires elementwise multiplication (denoted by $$\odot$$). Using the chain rule through the hidden layer output $$\mathbf{h}$$, the gradient is calculated as: $$ \frac{\partial J}{\partial \mathbf{z}} = 	extrm{prod}\left(\frac{\partial J}{\partial \mathbf{h}}, \frac{\partial \mathbf{h}}{\partial \mathbf{z}}ight) = \frac{\partial J}{\partial \mathbf{h}} \odot \phi'\left(\mathbf{z}ight) $$ where $$\phi'(\mathbf{z})$$ represents the local derivative of the activation function.

Gradient of Objective Function with Respect to Intermediate Variable

The elementwise multiplication operator, denoted mathematically by the symbol $$\odot$$, is used to multiply the corresponding elements of two tensors of identical shape. In neural networks, this operator is specifically required when applying the chain rule to calculate the gradient of an intermediate variable that has passed through a function that applies elementwise, such as an activation function.

Claude

An element-wise product (also known as the Hadamard product) of two matrices with the same shape, such as $$A_{m,n}$$ and $$B_{m,n}$$, results in a new matrix of identical shape. Each element in the resulting matrix is the product of the elements at the corresponding positions in matrices $$A$$ and $$B$$. This is a specific instance of a binary elementwise tensor operation.

Element-wise Product

Dive into Deep Learning

In linear algebra, the standard operation of matrix-matrix multiplication (frequently referred to simply as matrix multiplication) is distinct from the Hadamard product. While the Hadamard product involves the elementwise multiplication of two matrices of the exact same dimensions, standard matrix multiplication involves the dot products of rows and columns and requires the inner dimensions of the two matrices to align.

Learn Before

Related

Learn After