Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

https://paperswithcode.com/method/bert

Ohio State University, Columbus

University of Michigan - Ann Arbor

Bert is a famous and impressive word embedding model proposed by google in recent years. It's a milestone contribution in natural language processing area. Bert use mask language model to solve the problem of Elmo and OpenAI-GPT that could only use the forward direction information.(It seems that it only uses a naive weighted average for the vectors gotten by forward direction and backward direction in Elmo. It's not a good way to use this kind of information.) It nearly updates all the sota of the natural language processing and provides a universial pre-trained model for natural language processing area. 

Bert

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

 $\textbf{B}$idirectional $\textbf{E}$ncoder $\textbf{R}$epresentations from $\textbf{T}$ransformers
- A new language representation model that is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers
- Beneficial because it is a generalist approach that allows for customization to specific tasks (question answering, language inference, etc) by adding just one additional layer, while other approaches might build entirely different models for different language-processing tasks

Learn Before

Related