In the design of convolutional blocks, introducing a bottleneck is a common practice. For a given bottleneck ratio $$k_i \geq 1$$, the number of channels allocated within each block for stage $$i$$ is calculated as $$c_i/k_i$$, where $$c_i$$ represents the target number of output channels. However, empirical evaluations within the AnyNet design space reveal that employing a bottleneck ratio is not genuinely effective for improving performance; experiments show that a bottleneck ratio of $$k = 1$$ performs best, meaning bottlenecks should generally be omitted from the architecture.

Claude

The generic AnyNet design space provides $$17$$ configurable parameters, yielding a vast number of potential network configurations. These parameters determine the specific architecture of the network body and include: the block widths (number of channels) $$c_0, \dots, c_4$$, the depths (number of blocks per stage) $$d_1, \dots, d_4$$, the bottleneck ratios $$k_1, \dots, k_4$$ (where $$k_i \geq 1$$), and the group widths (number of groups for grouped convolutions) $$g_1, \dots, g_4$$.

AnyNet Design Space Parameters

Dive into Deep Learning

AnyNet Bottleneck Ratio Formula and Effectiveness

When configuring a neural network, identifying the single best parameter choice from a massive combination space (e.g., $$2^{17} = 131072$$ combinations) is computationally infeasible and offers little insight for future architectures. Furthermore, this approach is flawed because due to the stochasticity in training (e.g., rounding, shuffling, bit errors), no two runs are likely to produce exactly the same results. Instead, an effective strategy relies on the assumption that general design principles exist and many networks will perform well. By sampling uniformly from the space of configurations and evaluating their performance, researchers can analyze the distribution of accuracies to discover simple rules and general guidelines that relate parameter choices.

Network Design Space Distribution Strategy

The RegNet architecture emerges from the $$\textrm{AnyNetX}_E$$ design space, which simplifies the generic AnyNet design by enforcing a set of easily interpretable principles. Specifically, it requires sharing both the bottleneck ratio ($$k_i = k$$) and the group width ($$g_i = g$$) across all stages $$i$$. Furthermore, it mandates that both the network width ($$c_i$$) and the network depth ($$d_i$$) must increase monotonically across successive stages, meaning $$c_i \leq c_{i+1}$$ and $$d_i \leq d_{i+1}$$.

Learn Before

Related