University of Michigan - Ann Arbor

To examine how layers transform intermediate representations on CIFAR-10, layer responses are tracked across network architectures. Specifically, the responses are captured as the outputs of each $3 \times 3$ convolutional layer immediately after Batch Normalization (BN) and prior to any subsequent non-linearity, such as ReLU activation or element-wise addition.

The response strength is quantified by calculating the standard deviation (std) of these pre-activation signals across layers. In residual networks, these measurements reflect the actual response magnitude of the learned residual function $\mathcal{F}(\mathbf{x})$ rather than the total combined output $\mathcal{F}(\mathbf{x}) + \mathbf{x}$.

Learn Before

Related