In optimal RegNet configurations, the width of the network (number of channels) increases linearly with the block index. This relationship is approximated by the equation $$c_j \approx c_0 + c_a j$$, where $$j$$ represents the block index and $$c_a > 0$$ is the slope. Because the block width can only be changed at the beginning of each stage rather than per block, this continuous linear scaling is practically implemented as a piecewise constant function engineered to match the theoretical linear dependence.

RegNet Channel Width Scaling

The RegNetX32 is an effective 32-layer architectural variant derived from the RegNet design space. It is configured without a bottleneck ($$k = 1$$) and uses a uniform group width of $$g = 16$$ across all stages. For its stage progression, the first stage is $$d_1=4$$ blocks deep with a width of $$c_1 = 32$$ channels, and the second stage is $$d_2=6$$ blocks deep with a width of $$c_2 = 80$$ channels. As with all RegNetX models, each stage progressively reduces the spatial resolution while increasing the number of output channels.

RegNetX32 Variant

The fundamental design principles of the RegNet space—such as shared bottleneck ratios, shared group widths, and linear channel scaling—are robust enough to apply beyond standard convolutional blocks. They remain highly effective when applied to Squeeze-and-Excitation (SE) network designs, which utilize a global channel activation mechanism. This specific adaptation of the RegNet principles with SE blocks is known as the RegNetY architecture.

RegNetY Architecture

The RegNet architecture emerges from the $$\textrm{AnyNetX}_E$$ design space, which simplifies the generic AnyNet design by enforcing a set of easily interpretable principles. Specifically, it requires sharing both the bottleneck ratio ($$k_i = k$$) and the group width ($$g_i = g$$) across all stages $$i$$. Furthermore, it mandates that both the network width ($$c_i$$) and the network depth ($$d_i$$) must increase monotonically across successive stages, meaning $$c_i \leq c_{i+1}$$ and $$d_i \leq d_{i+1}$$.

Claude

The generic AnyNet design space provides $$17$$ configurable parameters, yielding a vast number of potential network configurations. These parameters determine the specific architecture of the network body and include: the block widths (number of channels) $$c_0, \dots, c_4$$, the depths (number of blocks per stage) $$d_1, \dots, d_4$$, the bottleneck ratios $$k_1, \dots, k_4$$ (where $$k_i \geq 1$$), and the group widths (number of groups for grouped convolutions) $$g_1, \dots, g_4$$.

AnyNet Design Space Parameters

Dive into Deep Learning

In the design of convolutional blocks, introducing a bottleneck is a common practice. For a given bottleneck ratio $$k_i \geq 1$$, the number of channels allocated within each block for stage $$i$$ is calculated as $$c_i/k_i$$, where $$c_i$$ represents the target number of output channels. However, empirical evaluations within the AnyNet design space reveal that employing a bottleneck ratio is not genuinely effective for improving performance; experiments show that a bottleneck ratio of $$k = 1$$ performs best, meaning bottlenecks should generally be omitted from the architecture.

AnyNet Bottleneck Ratio Formula and Effectiveness

When configuring a neural network, identifying the single best parameter choice from a massive combination space (e.g., $$2^{17} = 131072$$ combinations) is computationally infeasible and offers little insight for future architectures. Furthermore, this approach is flawed because due to the stochasticity in training (e.g., rounding, shuffling, bit errors), no two runs are likely to produce exactly the same results. Instead, an effective strategy relies on the assumption that general design principles exist and many networks will perform well. By sampling uniformly from the space of configurations and evaluating their performance, researchers can analyze the distribution of accuracies to discover simple rules and general guidelines that relate parameter choices.

Learn Before

Related

Learn After