The Network in Network (NiN) block can be implemented programmatically across deep learning frameworks as a sequential container of convolutional layers. It consists of an initial spatial convolution followed by two $$1 \times 1$$ convolutions that act as per-pixel fully connected layers. Each convolutional layer in the block is immediately followed by a ReLU activation function.

**PyTorch Implementation:**
```python
def nin_block(out_channels, kernel_size, strides, padding):
    return nn.Sequential(
        nn.LazyConv2d(out_channels, kernel_size, strides, padding), nn.ReLU(),
        nn.LazyConv2d(out_channels, kernel_size=1), nn.ReLU(),
        nn.LazyConv2d(out_channels, kernel_size=1), nn.ReLU())
```

**MXNet Implementation:**
```python
def nin_block(num_channels, kernel_size, strides, padding):
    blk = nn.Sequential()
    blk.add(nn.Conv2D(num_channels, kernel_size, strides, padding,
                      activation='relu'),
            nn.Conv2D(num_channels, kernel_size=1, activation='relu'),
            nn.Conv2D(num_channels, kernel_size=1, activation='relu'))
    return blk
```

**JAX Implementation:**
```python
def nin_block(out_channels, kernel_size, strides, padding):
    return nn.Sequential([
        nn.Conv(out_channels, kernel_size, strides, padding),
        nn.relu,
        nn.Conv(out_channels, kernel_size=(1, 1)), nn.relu,
        nn.Conv(out_channels, kernel_size=(1, 1)), nn.relu])
```

**TensorFlow Implementation:**
```python
def nin_block(out_channels, kernel_size, strides, padding):
    return tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(out_channels, kernel_size, strides=strides,
                           padding=padding),
    tf.keras.layers.Activation('relu'),
    tf.keras.layers.Conv2D(out_channels, 1),
    tf.keras.layers.Activation('relu'),
    tf.keras.layers.Conv2D(out_channels, 1),
    tf.keras.layers.Activation('relu')])
```

NiN Block Code Implementation

The NiN block is the core structural component of the Network in Network architecture. It presents a key structural difference from standard convolutional blocks, such as those found in the VGG network. While a VGG block retains standard spatial convolutions (e.g., $$3 	imes 3$$), a NiN block follows an initial spatial convolution immediately with a sequence of $$1 	imes 1$$ convolutions. By relying on these $$1 	imes 1$$ convolutions to act as per-pixel fully connected layers within the blocks, the NiN architecture eliminates the need for a giant, parameter-heavy fully connected layer at the end of the network.

Claude

The Network in Network (NiN) architecture introduces the concept of applying a fully connected layer at each individual pixel location (height and width) of a feature map. Convolutional layers output four-dimensional tensors corresponding to the example, channel, height, and width, whereas traditional fully connected layers expect two-dimensional tensors corresponding to the example and feature. To bridge this gap without flattening the spatial dimensions prematurely, NiN utilizes $$1 	imes 1$$ convolutions. A $$1 	imes 1$$ convolution effectively acts as a fully connected layer operating independently on each pixel's channels across the spatial grid.

Network in Network (NiN) Architecture

Dive into Deep Learning

NiN Block

The macro-architecture of the Network in Network (NiN) model is constructed by chaining multiple NiN blocks, using the same initial convolution sizes and output channels as AlexNet (specifically $$11	imes 11$$, $$5	imes 5$$, and $$3	imes 3$$). Each NiN block within the network is followed by a max-pooling layer with a stride of $$2$$ and a window shape of $$3	imes 3$$ to reduce spatial dimensions. A significant design difference between the NiN model and previous architectures like AlexNet or VGG is that NiN completely avoids the use of traditional fully connected layers.

Learn Before

Related

Learn After