For a standard Transformer-based language model with approximately 110 million total parameters, what are the specific values for the following three architectural hyperparameters: number of Transformer layers, number of attention heads, and hidden size?

Google

The BERT-base model is configured with a specific set of hyperparameters that determine its overall size and architectural capacity. These key settings include a hidden size ($$d$$) of 768, a model depth of 12 Transformer layers ($$L$$), and 12 attention heads ($$n_{\mathrm{head}}$$). Together, this structural configuration produces a model containing a total of 110 million parameters.

BERT-base Hyperparameters

A standard language model architecture with approximately 110 million parameters is built using a specific combination of layers, hidden size, and attention heads. Which of the following configurations correctly represents this model?

Hyperparameter Configuration for a Standard Language Model

A standard language model architecture with approximately 110 million parameters is defined by a specific set of hyperparameters. Match each hyperparameter with its correct value for this model.

Learn Before

Related