Convolutional transpose layers, also known as transposed convolutions, are neural network layers used to upsample the spatial dimensions of an input tensor. While a standard convolutional layer with a stride of 2 halves both the height and the width of its input, a convolutional transpose layer with a stride of 2 doubles them. These layers are widely used in generative and decoder architectures, such as autoencoders, to reconstruct spatial resolution from compressed latent representations.

San Diego State University

Google

In contrast to a regular convolution that reduces input elements via a kernel, the transposed convolution broadcasts input elements via the kernel, producing an output that is generally larger than the input. For a basic transposed convolution with a stride of $$1$$ and no padding, an $$n_h 	imes n_w$$ input tensor and a $$k_h 	imes k_w$$ kernel are used to produce an $$(n_h + k_h - 1) 	imes (n_w + k_w - 1)$$ output tensor. For each element in the input tensor, it is multiplied by the entire kernel to produce a $$k_h 	imes k_w$$ tensor. This resulting tensor replaces a corresponding portion of an intermediate tensor initialized with zeros, based on the element's original position. The final output is formed by summing all these intermediate results. For example, if both the input tensor and the kernel are $$2 	imes 2$$ matrices with values $$[[0, 1], [2, 3]]$$, the basic transposed convolution yields a $$3 	imes 3$$ output tensor with values $$[[0, 0, 1], [0, 4, 6], [4, 12, 9]]$$.

Basic Transposed Convolution Operation

Transposed convolution handles multiple input and output channels using the same fundamental principles as regular convolution. If the input tensor has $$c_i$$ channels, the transposed convolution applies a distinct $$k_h 	imes k_w$$ spatial kernel to each individual input channel. When configuring the layer to produce multiple output channels, it allocates a complete set of these $$c_i$$ kernels for every desired output channel. As a result, the complete multi-channel kernel tensor possesses a total shape of $$c_i 	imes k_h 	imes k_w$$ for each output channel, allowing the network to learn rich, multi-dimensional upsampling transformations.

Multiple Channels in Transposed Convolution

Unlike regular convolution, where padding is added to the borders of the input tensor, padding in transposed convolution is applied directly to the output. Specifying a padding value removes that number of rows and columns from the outer boundaries of the transposed convolution result. For example, if we apply a padding of $$1$$ on all sides, the first and last rows and columns are discarded from the final output tensor. In frameworks like PyTorch and MXNet, this can be implemented using the `padding=1` argument in `nn.ConvTranspose2d` or `nn.Conv2DTranspose`.

```python
# PyTorch
tconv = nn.ConvTranspose2d(1, 1, kernel_size=2, padding=1, bias=False)
tconv.weight.data = K
tconv(X)
```

```python
# MXNet
tconv = nn.Conv2DTranspose(1, kernel_size=2, padding=1)
tconv.initialize(init.Constant(K))
tconv(X)
```

Padding in Transposed Convolution

In transposed convolution, strides determine the spatial arrangement of the intermediate results rather than the sliding movement of the kernel over the input. Specifically, a stride greater than $$1$$ increases the spacing between the broadcasted elements in the intermediate tensor, which expands both the height and width of the final output. For instance, increasing the stride from $$1$$ to $$2$$ with a $$2 \times 2$$ kernel places the generated values two pixels apart in the intermediate matrix. This expansion mechanism can be implemented by setting the stride parameter to `stride=2` in PyTorch's `nn.ConvTranspose2d` or `strides=2` in MXNet's `nn.Conv2DTranspose`.

```python
# PyTorch
tconv = nn.ConvTranspose2d(1, 1, kernel_size=2, stride=2, bias=False)
tconv.weight.data = K
tconv(X)
```

```python
# MXNet
tconv = nn.Conv2DTranspose(1, kernel_size=2, strides=2)
tconv.initialize(init.Constant(K))
tconv(X)
```

Strides in Transposed Convolution

A primary application of transposed convolutional layers is to perfectly reverse the spatial dimensionality transformations introduced by standard convolutional layers. If an input tensor $$\mathsf{X}$$ is passed through a convolutional layer $$f$$ to produce an output $$\mathsf{Y} = f(\mathsf{X})$$, a transposed convolutional layer $$g$$ can be constructed to invert this shape change. By configuring $$g$$ with the exact same spatial hyperparameters (kernel size, padding, and stride) as $$f$$, and setting its output channel count to match the channels of $$\mathsf{X}$$, the resulting tensor $$g(\mathsf{Y})$$ will possess the identical shape as the original input $$\mathsf{X}$$. This property can be verified programmatically in frameworks like PyTorch and MXNet by ensuring `tconv(conv(X)).shape == X.shape` when both layers share identical parameters.

```python
# PyTorch
X = torch.rand(size=(1, 10, 16, 16))
conv = nn.Conv2d(10, 20, kernel_size=5, padding=2, stride=3)
tconv = nn.ConvTranspose2d(20, 10, kernel_size=5, padding=2, stride=3)
tconv(conv(X)).shape == X.shape
```

```python
# MXNet
X = np.random.uniform(size=(1, 10, 16, 16))
conv = nn.Conv2D(20, kernel_size=5, padding=2, strides=3)
tconv = nn.Conv2DTranspose(10, kernel_size=5, padding=2, strides=3)
conv.initialize()
tconv.initialize()
tconv(conv(X)).shape == X.shape
```

Learn Before

Related