Google

The architecture of a Transformer encoder is defined by several essential hyperparameters. These include the vocabulary size ($$|V|$$) and the embedding size ($$d_e$$) used for token representations. Additionally, the hidden size ($$d$$) specifies the input and output dimensionality for both the self-attention and the Feed-Forward Network (FFN) sub-layers. Other crucial hyperparameters are the number of attention heads ($$n_{\mathrm{head}}$$) for the multi-head self-attention mechanism, the internal hidden layer size of the FFN ($$d_{\textrm{ffn}}$$), and the model depth ($$L$$), which indicates the number of stacked layers.

Key Hyperparameters of a Transformer Encoder

In Transformer architectures, the hidden size, denoted as $$d$$, specifies the dimensionality of the input and output vectors for each sub-layer. Furthermore, the majority of the internal hidden states generated within these sub-layers are also $$d$$-dimensional vectors. Because it determines the size of these internal representations, $$d$$ can generally be interpreted as a measure of the overall width of the network.

Hidden Size in Transformer Models

A machine learning engineer is designing a Transformer encoder for a complex language task. Their primary goal is to improve the model's ability to capture diverse and varied contextual relationships (e.g., syntactic, semantic, co-reference) from different parts of the input sequence simultaneously. Which hyperparameter adjustment would most directly address this specific goal?

An engineer is trying to increase the capacity of a Transformer encoder to better handle a complex language task. They are considering two options:

Option A: Double the hidden size ($d$).
Option B: Double the number of attention heads ($n_{head}$).

Compare these two options. Explain the likely impact of each on the model's learning capabilities and computational cost. Which option would you recommend if the primary goal is to allow the model to focus on more varied and nuanced relationships within the input text? Justify your choice.

Hyperparameter Tuning Trade-offs

An engineer is configuring a Transformer encoder. Match each key hyperparameter to its specific architectural role.

The Feed-Forward Network (FFN) sub-layers within Transformer models feature a hidden layer with a specific size denoted as $$d_{\textrm{ffn}}$$. This dimension is typically designed to be larger than the standard hidden size, $$d$$. A common architectural setup sets $$d_{\textrm{ffn}} = 4d$$. For more recent, larger-scale Transformers, $$d_{\textrm{ffn}}$$ can be assigned to an even larger value to boost capacity.

FFN Hidden Size in Transformers

In Transformer models, the vocabulary size, denoted as $$|V|$$, specifies the number of distinct tokens the model can recognize. Each input token corresponds to a specific entry in this vocabulary $$V$$. Choosing the size of this vocabulary involves a clear trade-off: a larger vocabulary allows the model to cover more surface form variations of words, but it simultaneously increases the overall storage requirements and parameter count of the model.

Vocabulary Size in Transformers

The expressive power of Transformer networks can be effectively enhanced by increasing the model depth, denoted by $$L$$, which represents the total number of stacked processing layers. In standard BERT architectures, the depth $$L$$ is typically configured to either 12 or 24. However, employing networks with even greater depth is a viable strategy to achieve further performance enhancements.

Model Depth in Transformers

When configuring multi-head self-attention sub-layers in Transformers, one must specify the number of heads, denoted as $$n_{\mathrm{head}}$$. Increasing this hyperparameter expands the number of distinct subspaces over which attention is computed. In practical implementations, it is common to configure the model such that $$n_{\mathrm{head}} \ge 4$$.

Number of Attention Heads

In Transformer models, the embedding size, denoted as $$d_e$$, defines the dimensionality of the real-valued vectors used to represent each token. As such, the final input vector for any given token is a $$d_e$$-dimensional real-valued vector. This vector is formed by summing its constituent parts—the token embedding, positional embedding, and segment embedding—each of which is independently a $$d_e$$-dimensional real-valued vector.

Learn Before

Related