We can construct a multi-output channel convolution kernel programmatically by concatenating existing kernel tensors. For example, if K is a multi-input channel kernel tensor for a single output channel, we can stack K, K + 1, and K + 2 along dimension 0 to create a kernel with three output channels. If the original K had a shape of $$ 2 \times 2 \times 2 $$ (representing $$ c_{\textrm{i}} \times k_{\textrm{h}} \times k_{\textrm{w}} $$), the new stacked kernel will have an overall shape of $$ 3 \times 2 \times 2 \times 2 $$.

Claude

Google

To produce an output with multiple channels ($$c_	extrm{o}$$), a kernel tensor of shape $$c_	extrm{i} 	imes k_	extrm{h} 	imes k_	extrm{w}$$ must be created for every output channel. These individual kernel tensors are then concatenated along the output channel dimension. This concatenation results in an overall convolution kernel shape of $$c_	extrm{o} 	imes c_	extrm{i} 	imes k_	extrm{h} 	imes k_	extrm{w}$$.

Multi-Output Channel Convolution Kernel Structure

Dive into Deep Learning

In a multi-output channel cross-correlation, the result for each individual output channel is calculated independently from the other output channels, using its corresponding $$c_	extrm{i} 	imes k_	extrm{h} 	imes k_	extrm{w}$$ slice from the convolution kernel and the entire multi-channel input tensor. Specifically, the kernel slice performs a multi-channel cross-correlation with all $$c_	extrm{i}$$ channels of the input tensor to produce one channel of the output. While the computation for each channel is independent, the multiple output channels do not learn representations independently; rather, they are optimized jointly, meaning specific directions in the channel space may correspond to useful feature detections, such as edge detectors.

Multi-Output Channel Cross-Correlation Operation

In a convolutional layer, breaking up a standard convolution from $$c_\textrm{i}$$ input channels to $$c_\textrm{o}$$ output channels into $$g$$ independent groups is known as grouped convolution. In this operation, the input channels are divided into $$g$$ groups of size $$c_\textrm{i}/g$$, which independently generate $$g$$ outputs of size $$c_\textrm{o}/g$$. This approach proportionally reduces the computational cost from $$\mathcal{O}(c_\textrm{i} \cdot c_\textrm{o})$$ to $$\mathcal{O}(c_\textrm{i} \cdot c_\textrm{o} / g)$$, making it $$g$$ times faster. Additionally, the number of parameters required for the operation decreases from a single $$c_\textrm{i} \times c_\textrm{o}$$ matrix to $$g$$ smaller matrices of size $$(c_\textrm{i}/g) \times (c_\textrm{o}/g)$$, yielding a $$g$$ times reduction in parameters. It is assumed that both $$c_\textrm{i}$$ and $$c_\textrm{o}$$ are fully divisible by $$g$$.

Learn Before

Related