Transfer Learning Method 1

Transfer Learning Method 2

A simple method for designing a self-supervised classification task to train an encoder is Next Sentence Prediction (NSP), as presented in the original BERT paper. This approach is built on the assumption that a good text encoder should effectively capture the relationship between two sentences. To model this, NSP uses the output of encoding two consecutive sentences, $$\mathrm{Sent}_{A}$$ and $$\mathrm{Sent}_{B}$$, to determine whether $$\mathrm{Sent}_{B}$$ is indeed the next sentence following $$\mathrm{Sent}_{A}$$. For example, if $$\mathrm{Sent}_{A}$$ is 'It is raining .' and $$\mathrm{Sent}_{B}$$ is 'I need an umbrella .', the model is tasked with recognizing this sequential relationship.

Next Sentence Prediction (NSP)

During the supervised learning phase for downstream tasks, the fine-tuning process for BERT shares two key similarities with GPT. First, the contextual representations generated by the pre-trained Transformer encoder are fed into an added output layer, requiring minimal modifications to the core architecture regardless of the task's nature (e.g., predicting a label for every token versus the entire sequence). Second, all parameters of the pre-trained model undergo fine-tuning, while the new parameters of the additional output layer are trained from scratch.

State University of New York at Stony Brook

Claude

 $\textbf{B}$idirectional $\textbf{E}$ncoder $\textbf{R}$epresentations from $\textbf{T}$ransformers
- A new language representation model that is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers
- Beneficial because it is a generalist approach that allows for customization to specific tasks (question answering, language inference, etc) by adding just one additional layer, while other approaches might build entirely different models for different language-processing tasks

What is BERT?

The GPT (Generative Pre-Training) model represents an effort to design a general, task-agnostic architecture for context-sensitive representations. Built on a Transformer decoder, it pretrains a language model to represent text sequences. When adapted for downstream applications, the model's output feeds directly into an added linear layer to predict task labels. In sharp contrast to earlier models that freeze their pretrained weights, GPT fine-tunes all parameters in the pretrained Transformer decoder during supervised learning. Evaluated on twelve tasks encompassing natural language inference, question answering, sentence similarity, and classification, GPT improved the state of the art in nine of them with minimal changes to its core architecture.

GPT (Generative Pre-Training)

Dive into Deep Learning

BERT models are built to represent either individual sentences or pairs of sentences, equipping them to tackle a variety of downstream language understanding problems. When a task requires handling two sentences simultaneously, the input is arranged as a single combined sequence containing both sentences, typically denoted as $$\mathrm{Sent}_{A}$$ and $$\mathrm{Sent}_{B}$$.

BERT Input Representation: Single and Paired Sentences

BERT is recognized as a milestone model in NLP that has inspired significant subsequent research. Its key contributions include demonstrating the importance of bidirectional pre-training for language representations, which was enabled by the masked language model. BERT also showed that pre-trained representations reduce the need for heavily-engineered, task-specific architectures. It was the first fine-tuning based representation model to achieve state-of-the-art performance on a wide range of sentence-level and token-level tasks, advancing the state of the art for eleven NLP tasks.

BERT's Contributions and Impact

As proposed in the original paper by Devlin et al. (2019), the standard BERT model is a Transformer encoder pre-trained with a dual-task objective. This training process involves simultaneously learning from two tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). The total training loss is calculated as the sum of the individual losses from these two objectives.

Training Objective of the Standard BERT Model

The architectures of ELMo, GPT, and BERT offer distinct approaches to modeling context for downstream adaptation. ELMo provides bidirectional context encoding but requires customized, task-specific architectures for different applications. In contrast, GPT employs a task-agnostic approach that uses the same architecture across tasks, but it is limited to left-to-right context encoding. BERT unites the best of both methodologies: by utilizing a pretrained Transformer encoder, it achieves deep bidirectional context representations while remaining entirely task-agnostic, allowing it to be adapted to a wide variety of tasks with minimal structural changes.

Comparison of ELMo, GPT, and BERT

BERT significantly advanced the state of the art across eleven distinct natural language processing tasks. These tasks fall into four broad categories: single text classification (such as sentiment analysis), text pair classification (such as natural language inference), question answering, and text tagging (such as named entity recognition). By replacing complex, task-specific architectures with a simple, fine-tunable bidirectional pre-trained model, BERT revolutionized solutions across these varied language applications.

BERT Performance Improvements on NLP Tasks

Similarities Between BERT and GPT in Fine-Tuning

A foundational generative language model introduced in 2018 significantly improved the ability to capture relationships between words far apart in a text, a major challenge for previous sequential models. Which of the following best analyzes the core architectural innovation responsible for this leap in performance?

A pioneering 2018 language model was based on a transformer architecture that processed text strictly in a left-to-right sequence to predict the next word. Evaluate the primary conceptual limitation of this unidirectional approach. In your analysis, discuss specific types of language understanding tasks where this design choice would likely result in suboptimal performance and explain why.

Critique of an Early Transformer-Based Language Model

A pioneering generative language model from 2018, which was the first to be based on the transformer architecture, was trained using a specific objective on a large corpus of unlabeled text. In your own words, describe the primary goal of this model's pre-training process.

Training Objective of an Early Transformer Model

Introduced a year after its predecessor, GPT-2 is a significantly larger Transformer-decoder language model containing $$1.5$$ billion parameters and pretrained on $$40$$ GB of text. It introduced architectural refinements such as pre-normalization, as well as improved initialization and weight-scaling. GPT-2 was groundbreaking for achieving state-of-the-art results on language modeling benchmarks and promising results on multiple other tasks without requiring any parameter updates or architectural modifications.

GPT-2

Due to the autoregressive nature of its language modeling objective, the Generative Pre-Training (GPT) model processes text strictly in a forward, left-to-right direction. Consequently, a word's representation is determined solely by the context to its left. For example, in the phrases 'i went to the bank to deposit cash' and 'i went to the bank to sit down', the left context for the word 'bank' is identical in both cases. Because GPT cannot look ahead to the rightward context that differentiates the intended meaning, it returns the exact same representation for 'bank' in both sentences, exposing a critical limitation in its context sensitivity.

Learn Before

Related

Learn After