- This paper demonstrates the importance of bidirectional pre-training for language representations
- Masked language model enables bidirectional pre-training, where previous models used unidirectional language models for pre-training
- It shows that pre-trained representations reduce the need for many heavily-engineered task-specific architectures
- BERT is the first fine-tuning based representation model that achieves state-of-the-art performance on a large suite of sentence-level and token-level tasks
- BERT advances the state of the art for eleven NLP tasks


Ohio State University, Columbus

 $\textbf{B}$idirectional $\textbf{E}$ncoder $\textbf{R}$epresentations from $\textbf{T}$ransformers
- A new language representation model that is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers
- Beneficial because it is a generalist approach that allows for customization to specific tasks (question answering, language inference, etc) by adding just one additional layer, while other approaches might build entirely different models for different language-processing tasks

What is BERT?

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

https://paperswithcode.com/method/bert

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

This study improves the fine-tuning based approaches by proposing BERT by:

- Using a masked language model (MLM) to alleviate the unidirectionality constraint
- Using next sentence prediction to jointly pre-train text-pair representations

BERT’s Innovations

BERT's Contributions

This paper presents BERT fine-tuning results on 11 Natural Language Processing (NLP) tasks.

BERT Experiments

Input representation of BERT is  3 different input embeddings merged.

Learn Before

Related