In the design of convolutional blocks, introducing a bottleneck is a common practice. For a given bottleneck ratio $$k_i \geq 1$$, the number of channels allocated within each block for stage $$i$$ is calculated as $$c_i/k_i$$, where $$c_i$$ represents the target number of output channels. However, empirical evaluations within the AnyNet design space reveal that employing a bottleneck ratio is not genuinely effective for improving performance; experiments show that a bottleneck ratio of $$k = 1$$ performs best, meaning bottlenecks should generally be omitted from the architecture.

AnyNet Bottleneck Ratio Formula and Effectiveness

When configuring a neural network, identifying the single best parameter choice from a massive combination space (e.g., $$2^{17} = 131072$$ combinations) is computationally infeasible and offers little insight for future architectures. Furthermore, this approach is flawed because due to the stochasticity in training (e.g., rounding, shuffling, bit errors), no two runs are likely to produce exactly the same results. Instead, an effective strategy relies on the assumption that general design principles exist and many networks will perform well. By sampling uniformly from the space of configurations and evaluating their performance, researchers can analyze the distribution of accuracies to discover simple rules and general guidelines that relate parameter choices.

Network Design Space Distribution Strategy

The RegNet architecture emerges from the $$\textrm{AnyNetX}_E$$ design space, which simplifies the generic AnyNet design by enforcing a set of easily interpretable principles. Specifically, it requires sharing both the bottleneck ratio ($$k_i = k$$) and the group width ($$g_i = g$$) across all stages $$i$$. Furthermore, it mandates that both the network width ($$c_i$$) and the network depth ($$d_i$$) must increase monotonically across successive stages, meaning $$c_i \leq c_{i+1}$$ and $$d_i \leq d_{i+1}$$.

RegNet Design Space

The generic AnyNet design space provides $$17$$ configurable parameters, yielding a vast number of potential network configurations. These parameters determine the specific architecture of the network body and include: the block widths (number of channels) $$c_0, \dots, c_4$$, the depths (number of blocks per stage) $$d_1, \dots, d_4$$, the bottleneck ratios $$k_1, \dots, k_4$$ (where $$k_i \geq 1$$), and the group widths (number of groups for grouped convolutions) $$g_1, \dots, g_4$$.

Claude

The body of an AnyNet architecture conducts the majority of feature extraction by operating on images at progressively decreasing resolutions. Typically designed to process ImageNet inputs (e.g., reducing a $$224 \times 224 \times 3$$ image to a $$7 \times 7 \times c_4$$ representation), the body is divided into multiple stages (e.g., $$4$$ stages). Each stage quarters the total spatial resolution (by halving both height and width) and is constructed from a sequence of ResNeXt blocks. The initial block in a stage applies a stride of $$2$$ to downsample the feature map, which necessitates a $$1 \times 1$$ convolution on the residual branch to match dimensions, while all subsequent blocks in the stage maintain constant resolution and channel counts.

AnyNet Body

Dive into Deep Learning

AnyNet Design Space Parameters

An AnyNet stage is implemented by sequentially stacking a specified depth of ResNeXt blocks. The first block in the sequence uses a spatial stride of $$2$$ and a $$1 \times 1$$ convolution on the residual branch to halve the feature map resolution. The remaining blocks use a default stride of $$1$$, preserving the spatial dimensions.

PyTorch Implementation:
```python
@d2l.add_to_class(AnyNet)
def stage(self, depth, num_channels, groups, bot_mul):
    blk = []
    for i in range(depth):
        if i == 0:
            blk.append(d2l.ResNeXtBlock(num_channels, groups, bot_mul,
                use_1x1conv=True, strides=2))
        else:
            blk.append(d2l.ResNeXtBlock(num_channels, groups, bot_mul))
    return nn.Sequential(*blk)
```

MXNet Implementation:
```python
@d2l.add_to_class(AnyNet)
def stage(self, depth, num_channels, groups, bot_mul):
    net = nn.Sequential()
    for i in range(depth):
        if i == 0:
            net.add(d2l.ResNeXtBlock(
                num_channels, groups, bot_mul, use_1x1conv=True, strides=2))
        else:
            net.add(d2l.ResNeXtBlock(
                num_channels, num_channels, groups, bot_mul))
    return net
```

JAX Implementation:
```python
@d2l.add_to_class(AnyNet)
def stage(self, depth, num_channels, groups, bot_mul):
    blk = []
    for i in range(depth):
        if i == 0:
            blk.append(d2l.ResNeXtBlock(num_channels, groups, bot_mul,
                use_1x1conv=True, strides=(2, 2), training=self.training))
        else:
            blk.append(d2l.ResNeXtBlock(num_channels, groups, bot_mul,
                                        training=self.training))
    return nn.Sequential(blk)
```

TensorFlow Implementation:
```python
@d2l.add_to_class(AnyNet)
def stage(self, depth, num_channels, groups, bot_mul):
    net = tf.keras.models.Sequential()
    for i in range(depth):
        if i == 0:
            net.add(d2l.ResNeXtBlock(num_channels, groups, bot_mul,
                use_1x1conv=True, strides=2))
        else:
            net.add(d2l.ResNeXtBlock(num_channels, groups, bot_mul))
    return net
```

Learn Before

Related

Learn After