Describe the trade-offs of applying label smoothing during Transformer training. Discuss how label smoothing alters the model's behavior, its impact on perplexity, and its effect on final task metrics like BLEU score and accuracy.
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.2 Transformer Training and Evaluation - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Transformer Training, Regularization, and Optimization - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Related
Where is residual dropout applied within each sub-layer of the Transformer architecture?
In the Transformer architecture, dropout is applied to the sums of the embeddings and positional encodings in both the encoder and decoder stacks.
What is the dropout rate () utilized in the base Transformer model?
Describe the trade-offs of applying label smoothing during Transformer training. Discuss how label smoothing alters the model's behavior, its impact on perplexity, and its effect on final task metrics like BLEU score and accuracy.