Instead of relying on parameter-heavy fully connected layers for final classification, the Network in Network (NiN) architecture utilizes a global average pooling layer. The final NiN block is configured to produce a number of output channels that exactly equals the target number of label classes. A global average pooling layer is then applied to these feature maps, which yields a vector of logits. This structural design significantly reduces the overall number of required model parameters, although it may potentially lead to an increase in training time.

Global Average Pooling in NiN

The programmatic implementation of the Network in Network (NiN) model instantiates the architecture using deep learning frameworks by chaining multiple NiN blocks, max-pooling layers, and a final global average pooling layer. The sequence begins with initial spatial convolutions followed by max-pooling to halve the spatial dimensions, mirroring AlexNet's initial stages. Instead of ending with parameter-heavy dense layers, the network concludes with a NiN block that outputs a channel for each target class, followed by global average pooling and a flattening operation to yield the final classification logits.

**PyTorch Implementation:**
```python
class NiN(d2l.Classifier):
    def __init__(self, lr=0.1, num_classes=10):
        super().__init__()
        self.save_hyperparameters()
        self.net = nn.Sequential(
            nin_block(96, kernel_size=11, strides=4, padding=0),
            nn.MaxPool2d(3, stride=2),
            nin_block(256, kernel_size=5, strides=1, padding=2),
            nn.MaxPool2d(3, stride=2),
            nin_block(384, kernel_size=3, strides=1, padding=1),
            nn.MaxPool2d(3, stride=2),
            nn.Dropout(0.5),
            nin_block(num_classes, kernel_size=3, strides=1, padding=1),
            nn.AdaptiveAvgPool2d((1, 1)),
            nn.Flatten())
        self.net.apply(d2l.init_cnn)
```

**MXNet Implementation:**
```python
class NiN(d2l.Classifier):
    def __init__(self, lr=0.1, num_classes=10):
        super().__init__()
        self.save_hyperparameters()
        self.net = nn.Sequential()
        self.net.add(
            nin_block(96, kernel_size=11, strides=4, padding=0),
            nn.MaxPool2D(pool_size=3, strides=2),
            nin_block(256, kernel_size=5, strides=1, padding=2),
            nn.MaxPool2D(pool_size=3, strides=2),
            nin_block(384, kernel_size=3, strides=1, padding=1),
            nn.MaxPool2D(pool_size=3, strides=2),
            nn.Dropout(0.5),
            nin_block(num_classes, kernel_size=3, strides=1, padding=1),
            nn.GlobalAvgPool2D(),
            nn.Flatten())
        self.net.initialize(init.Xavier())
```

**JAX Implementation:**
```python
class NiN(d2l.Classifier):
    lr: float = 0.1
    num_classes = 10
    training: bool = True

    def setup(self):
        self.net = nn.Sequential([
            nin_block(96, kernel_size=(11, 11), strides=(4, 4), padding=(0, 0)),
            lambda x: nn.max_pool(x, (3, 3), strides=(2, 2)),
            nin_block(256, kernel_size=(5, 5), strides=(1, 1), padding=(2, 2)),
            lambda x: nn.max_pool(x, (3, 3), strides=(2, 2)),
            nin_block(384, kernel_size=(3, 3), strides=(1, 1), padding=(1, 1)),
            lambda x: nn.max_pool(x, (3, 3), strides=(2, 2)),
            nn.Dropout(0.5, deterministic=not self.training),
            nin_block(self.num_classes, kernel_size=(3, 3), strides=1, padding=(1, 1)),
            lambda x: nn.avg_pool(x, (5, 5)),  # global avg pooling
            lambda x: x.reshape((x.shape[0], -1))  # flatten
        ])
```

**TensorFlow Implementation:**
```python
class NiN(d2l.Classifier):
    def __init__(self, lr=0.1, num_classes=10):
        super().__init__()
        self.save_hyperparameters()
        self.net = tf.keras.models.Sequential([
            nin_block(96, kernel_size=11, strides=4, padding='valid'),
            tf.keras.layers.MaxPool2D(pool_size=3, strides=2),
            nin_block(256, kernel_size=5, strides=1, padding='same'),
            tf.keras.layers.MaxPool2D(pool_size=3, strides=2),
            nin_block(384, kernel_size=3, strides=1, padding='same'),
            tf.keras.layers.MaxPool2D(pool_size=3, strides=2),
            tf.keras.layers.Dropout(0.5),
            nin_block(num_classes, kernel_size=3, strides=1, padding='same'),
            tf.keras.layers.GlobalAvgPool2D(),
            tf.keras.layers.Flatten()])
```

NiN Model Code Implementation

The macro-architecture of the Network in Network (NiN) model is constructed by chaining multiple NiN blocks, using the same initial convolution sizes and output channels as AlexNet (specifically $$11	imes 11$$, $$5	imes 5$$, and $$3	imes 3$$). Each NiN block within the network is followed by a max-pooling layer with a stride of $$2$$ and a window shape of $$3	imes 3$$ to reduce spatial dimensions. A significant design difference between the NiN model and previous architectures like AlexNet or VGG is that NiN completely avoids the use of traditional fully connected layers.

Claude

The Network in Network (NiN) architecture introduces the concept of applying a fully connected layer at each individual pixel location (height and width) of a feature map. Convolutional layers output four-dimensional tensors corresponding to the example, channel, height, and width, whereas traditional fully connected layers expect two-dimensional tensors corresponding to the example and feature. To bridge this gap without flattening the spatial dimensions prematurely, NiN utilizes $$1 	imes 1$$ convolutions. A $$1 	imes 1$$ convolution effectively acts as a fully connected layer operating independently on each pixel's channels across the spatial grid.

Network in Network (NiN) Architecture

Dive into Deep Learning

The NiN block is the core structural component of the Network in Network architecture. It presents a key structural difference from standard convolutional blocks, such as those found in the VGG network. While a VGG block retains standard spatial convolutions (e.g., $$3 	imes 3$$), a NiN block follows an initial spatial convolution immediately with a sequence of $$1 	imes 1$$ convolutions. By relying on these $$1 	imes 1$$ convolutions to act as per-pixel fully connected layers within the blocks, the NiN architecture eliminates the need for a giant, parameter-heavy fully connected layer at the end of the network.

Learn Before

Related

Learn After