Hidden Size in Transformer Models

A machine learning engineer is designing a Transformer encoder for a complex language task. Their primary goal is to improve the model's ability to capture diverse and varied contextual relationships (e.g., syntactic, semantic, co-reference) from different parts of the input sequence simultaneously. Which hyperparameter adjustment would most directly address this specific goal?

Hyperparameter Tuning Trade-offs

An engineer is configuring a Transformer encoder. Match each key hyperparameter to its specific architectural role.

FFN Hidden Size in Transformers

Vocabulary Size in Transformers

Model Depth in Transformers

Number of Attention Heads

Embedding Size in Transformer Models