When evaluating the performance of a fully convolutional network in a semantic segmentation task, accuracy is determined at the pixel level rather than the image level. The accuracy calculation assesses the proportion of individual pixels across the entire input for which the model correctly predicted the semantic class. This means the overall metric reflects the correctness of the predicted classes for all pixels simultaneously, treating each spatial position as an independent classification instance.

Claude

A Fully Convolutional Network (FCN) is designed for dense prediction tasks, such as semantic segmentation. The model architecture begins with a standard Convolutional Neural Network (CNN) to extract image features. Next, it employs a $$1 	imes 1$$ convolutional layer to transform the number of channels to match the number of target classes. Finally, it uses a transposed convolutional layer to scale the height and width of the feature maps back to the dimensions of the original input image. The resulting output has the same spatial dimensions as the input, with each output channel representing the predicted classes for the pixel at the corresponding spatial position.

Fully Convolutional Network (FCN) Architecture

Dive into Deep Learning

When utilizing a pretrained convolutional neural network, such as ResNet-18, for feature extraction in a Fully Convolutional Network (FCN), the final classification layers must be removed. Specifically, the global average pooling layer and the fully connected layer are discarded because they collapse spatial dimensions and are unnecessary for dense pixel-level predictions. The remaining layers form the feature extraction backbone of the FCN, which produces feature maps with reduced spatial dimensions. For example, given an input with a height of $$320$$ and width of $$480$$, the forward propagation reduces the spatial dimensions by a factor of $$1/32$$, resulting in an output shape of $$10 \times 15$$. This extraction process can be implemented in frameworks like PyTorch or MXNet by explicitly selecting all layers except the final pooling and dense layers.

```python
# PyTorch
net = nn.Sequential(*list(pretrained_net.children())[:-2])
```

```python
# MXNet
net = nn.HybridSequential()
for layer in pretrained_net.features[:-2]:
    net.add(layer)
```

Feature Extraction in Fully Convolutional Networks

In a Fully Convolutional Network (FCN), after image features have been extracted by the backbone network, a $$1 \times 1$$ convolutional layer is applied. The purpose of this layer is to transform the number of output channels from the feature extractor to match the exact number of target classes (for example, $$21$$ classes for the Pascal VOC2012 dataset) without altering the spatial dimensions of the feature maps.

```python
# PyTorch
num_classes = 21
net.add_module('final_conv', nn.Conv2d(512, num_classes, kernel_size=1))
```

```python
# MXNet
num_classes = 21
net.add(nn.Conv2D(num_classes, kernel_size=1))
```

Channel Transformation in Fully Convolutional Networks

To restore the spatial dimensions of feature maps to the original input image size in a Fully Convolutional Network (FCN), a transposed convolutional layer is employed. If the spatial dimensions need to be increased by a factor of $$s$$, the transposed convolution is configured with a stride of $$s$$. To achieve the exact original dimensions, the padding is set to $$s/2$$ (assuming $$s/2$$ is an integer), and the height and width of the convolution kernel are set to $$2s$$. For instance, to upscale a feature map by $$32$$ times, the stride is $$32$$, the padding is $$16$$, and the kernel size is $$64$$.

```python
# PyTorch
net.add_module('transpose_conv', nn.ConvTranspose2d(num_classes, num_classes,
                                    kernel_size=64, padding=16, stride=32))
```

```python
# MXNet
net.add(nn.Conv2DTranspose(num_classes, kernel_size=64, padding=16, strides=32))
```

Transposed Convolution Configuration for FCN Upsampling

Accuracy Calculation in Fully Convolutional Networks

In training a fully convolutional network for semantic segmentation, the loss function is fundamentally similar to that used in standard image classification tasks. However, because the network predicts a distinct class for every individual pixel, these predictions are represented by the output channels of the final transposed convolutional layer. Consequently, when calculating the loss—typically a cross-entropy loss—the computation must explicitly specify the channel dimension to correctly evaluate the predicted probabilities against the ground-truth classes for each spatial position.

```python
# PyTorch
def loss(inputs, targets):
    return F.cross_entropy(inputs, targets, reduction='none').mean(1).mean(1)
```

```python
# MXNet
loss = gluon.loss.SoftmaxCrossEntropyLoss(axis=1)
```

Learn Before

Related