The GLUE benchmark is a collection of diverse natural language understanding tasks.

General Language Understanding Evaluation (GLUE)

The SQuAD v1.1 is a collection of 100k crowdsourced question/answer pairs.

Stanford Question Answering Dataset 
(SQuAD v1.1)

The SQuAD 2.0 task extends the SQuAD 1.1 problem definition by allowing for the possibility that no short answer exists in the provided paragraph, making the problem more realistic.

Stanford Question Answering Dataset 
(SQuAD v2.0) 

The SWAG dataset contains 113k sentence-pair completion examples that evaluate grounded common-sense inference.

Situations With Adversarial Generations (SWAG)


Ablation studies on BERT are attempts to understand the relative importance of the various pieces of BERT’s functionality by testing them in isolation.

BERT Ablation Studies

This paper presents BERT fine-tuning results on 11 Natural Language Processing (NLP) tasks.

Ohio State University, Columbus

 $\textbf{B}$idirectional $\textbf{E}$ncoder $\textbf{R}$epresentations from $\textbf{T}$ransformers
- A new language representation model that is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers
- Beneficial because it is a generalist approach that allows for customization to specific tasks (question answering, language inference, etc) by adding just one additional layer, while other approaches might build entirely different models for different language-processing tasks

What is BERT?

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

https://paperswithcode.com/method/bert

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

BERT Experiments

BERT&GPT and Fine Tuning

BERT models are built to represent either individual sentences or pairs of sentences, equipping them to tackle a variety of downstream language understanding problems. When a task requires handling two sentences simultaneously, the input is arranged as a single combined sequence containing both sentences, typically denoted as $$\mathrm{Sent}_{A}$$ and $$\mathrm{Sent}_{B}$$.

BERT Input Representation: Single and Paired Sentences

BERT is recognized as a milestone model in NLP that has inspired significant subsequent research. Its key contributions include demonstrating the importance of bidirectional pre-training for language representations, which was enabled by the masked language model. BERT also showed that pre-trained representations reduce the need for heavily-engineered, task-specific architectures. It was the first fine-tuning based representation model to achieve state-of-the-art performance on a wide range of sentence-level and token-level tasks, advancing the state of the art for eleven NLP tasks.

BERT's Contributions and Impact

As proposed in the original paper by Devlin et al. (2019), the standard BERT model is a Transformer encoder pre-trained with a dual-task objective. This training process involves simultaneously learning from two tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). The total training loss is calculated as the sum of the individual losses from these two objectives.

Learn Before

Related

Learn After