Learn Before
Analyze the effect of varying the number of attention heads () on Transformer performance based on the ablation experiments on the English-to-German development set (newstest2013). Detail how single-head attention, the baseline 8-head configuration, and an excessively high head count compare.
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.2 Transformer Training and Evaluation - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Machine Translation and Constituency Parsing Evaluation - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Related
According to the ablation experiments on the English-to-German development set, what was the effect of replacing sinusoidal positional encodings with learned positional embeddings?
In the Transformer ablation study on newstest2013, completely removing dropout () decreased translation performance.
What happened to translation performance in the ablation study when the key dimension () was reduced, and what does this finding suggest about representation learning?
Analyze the effect of varying the number of attention heads () on Transformer performance based on the ablation experiments on the English-to-German development set (newstest2013). Detail how single-head attention, the baseline 8-head configuration, and an excessively high head count compare.