The architectures of ELMo, GPT, and BERT offer distinct approaches to modeling context for downstream adaptation. ELMo provides bidirectional context encoding but requires customized, task-specific architectures for different applications. In contrast, GPT employs a task-agnostic approach that uses the same architecture across tasks, but it is limited to left-to-right context encoding. BERT unites the best of both methodologies: by utilizing a pretrained Transformer encoder, it achieves deep bidirectional context representations while remaining entirely task-agnostic, allowing it to be adapted to a wide variety of tasks with minimal structural changes.

Claude

 $\textbf{B}$idirectional $\textbf{E}$ncoder $\textbf{R}$epresentations from $\textbf{T}$ransformers
- A new language representation model that is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers
- Beneficial because it is a generalist approach that allows for customization to specific tasks (question answering, language inference, etc) by adding just one additional layer, while other approaches might build entirely different models for different language-processing tasks

What is BERT?

Dive into Deep Learning

BERT models are built to represent either individual sentences or pairs of sentences, equipping them to tackle a variety of downstream language understanding problems. When a task requires handling two sentences simultaneously, the input is arranged as a single combined sequence containing both sentences, typically denoted as $$\mathrm{Sent}_{A}$$ and $$\mathrm{Sent}_{B}$$.

BERT Input Representation: Single and Paired Sentences

BERT is recognized as a milestone model in NLP that has inspired significant subsequent research. Its key contributions include demonstrating the importance of bidirectional pre-training for language representations, which was enabled by the masked language model. BERT also showed that pre-trained representations reduce the need for heavily-engineered, task-specific architectures. It was the first fine-tuning based representation model to achieve state-of-the-art performance on a wide range of sentence-level and token-level tasks, advancing the state of the art for eleven NLP tasks.

BERT's Contributions and Impact

As proposed in the original paper by Devlin et al. (2019), the standard BERT model is a Transformer encoder pre-trained with a dual-task objective. This training process involves simultaneously learning from two tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). The total training loss is calculated as the sum of the individual losses from these two objectives.

Training Objective of the Standard BERT Model

Comparison of ELMo, GPT, and BERT

BERT significantly advanced the state of the art across eleven distinct natural language processing tasks. These tasks fall into four broad categories: single text classification (such as sentiment analysis), text pair classification (such as natural language inference), question answering, and text tagging (such as named entity recognition). By replacing complex, task-specific architectures with a simple, fine-tunable bidirectional pre-trained model, BERT revolutionized solutions across these varied language applications.

BERT Performance Improvements on NLP Tasks

During the supervised learning phase for downstream tasks, the fine-tuning process for BERT shares two key similarities with GPT. First, the contextual representations generated by the pre-trained Transformer encoder are fed into an added output layer, requiring minimal modifications to the core architecture regardless of the task's nature (e.g., predicting a label for every token versus the entire sequence). Second, all parameters of the pre-trained model undergo fine-tuning, while the new parameters of the additional output layer are trained from scratch.

Learn Before

Related