Google

In the pre-training phase, an encoder model is trained using a self-supervision objective like Masked Language Modeling. The process begins by converting a corrupted input sequence, where some tokens are masked, into a sequence of embeddings. This embedding sequence is then fed into the encoder, which generates contextual vector representations for all input tokens. Finally, these representations are passed to an output layer, such as a Softmax model, which is trained to reconstruct the original masked tokens.

Self-Supervised Pre-training of Encoders via Masked Language Modeling

Causal Language Modeling (CLM), also known as conventional language modeling, can be understood as a specific instance of Masked Language Modeling (MLM). In CLM, the prediction of a token at a given position is constrained by masking all subsequent tokens in the right-hand context. This forces the model to rely exclusively on the preceding left-hand context, making it a unidirectional process. In contrast, the general MLM approach is bidirectional because it uses all unmasked tokens—from both the left and right contexts—to predict a masked token within a sequence.

Comparison of Masked vs. Causal Language Modeling

In Masked Language Modeling, the masking process is formally defined for an input sequence $$\mathbf{x}=x_0...x_m$$. A set of positions, $$\mathcal{A}(\mathbf{x}) = \{i_1,...,i_{u}\}$, is selected to be masked. This procedure generates a modified token sequence, $$\bar{\mathbf{x}}$$, in which the token at each selected position in $$\mathcal{A}(\mathbf{x})$$ is replaced with a special $$[\mathrm{MASK}]$$ symbol.

Formal Definition of the Masking Process in MLM

To illustrate Masked Language Modeling, consider the sentence 'The early bird catches the worm'. A training instance is created by masking one or more tokens. For example, masking a single token might yield 'The early bird $$[\mathrm{MASK}]$$ the worm', where the model predicts 'catches'. Alternatively, masking multiple tokens, such as 'early' and 'worm' (e.g., at indices $${}i_1 = 2$$ and $${}i_2 = 6$$), yields the corrupted sequence 'The $$[\mathrm{MASK}]$$ bird catches the $$[\mathrm{MASK}]$$'. The model must then correctly predict the masked words based on context.

Example of Masked Language Modeling with Single and Multiple Masks

Given an original text sequence $$\mathbf{x}$$ and its corrupted version $$\bar{\mathbf{x}}$$, optimizing a model to predict $$\mathbf{x}$$ based on $$\bar{\mathbf{x}}$$ can be thought of as an autoencoding-like process. The fundamental training objective is to maximize the reconstruction probability $$\Pr(\mathbf{x}|\bar{\mathbf{x}})$$. However, because there is a simple position-wise alignment between the two sequences, an unmasked token in $$\bar{\mathbf{x}}$$ is the exact same as the token in $$\mathbf{x}$$ at the same position. Since there is no need to consider the prediction for these unmasked tokens, the training objective is simplified to only maximize the probabilities for the masked tokens.

Training Objective of Masked Language Modeling (MLM)

A significant drawback of Masked Language Modeling is its reliance on a special $$[\mathrm{MASK}]$$ token during the training phase. Because this artificial token is not present in natural text during testing or real-world inference, it creates a discrepancy between how the model is trained and how it operates in practice at test time.

Drawback of Masked Language Modeling: The [MASK] Token Discrepancy

A key limitation of the auto-encoding objective in Masked Language Modeling (MLM) is its failure to account for dependencies among the masked tokens. The model is trained to predict each masked token independently of the others. For example, if two tokens $$x_2$$ and $$x_6$$ in a sequence are masked, the prediction for the first masked token ($$x_2$$) is generated independently of the second masked token ($$x_6$$), even though $$x_6$$ should ideally be considered within the context of $$x_2$$.

Limitation of MLM: Ignoring Dependencies Between Masked Tokens

In the Replaced Token Detection framework, the generator is a small masked language model tasked with creating a corrupted version of the original input text. Its process involves two main steps: first, it randomly masks a subset of tokens in a sequence. Second, it is trained to predict the original tokens for these masked positions. The generator then outputs a new sequence where the masked tokens have been replaced by its predictions, which may or may not match the original tokens. This altered sequence is then passed to the discriminator.

The Generator in Replaced Token Detection

In Masked Language Modeling, the random selection of tokens for masking can result in the masking of consecutive tokens within a sequence [Joshi et al., 2020]. This means that two or more adjacent words may be replaced by `[MASK]` symbols in a single training instance.

Consecutive Token Masking in MLM

In the standard implementation of Masked Language Modeling for the BERT model, 15% of the tokens within each input sequence are randomly chosen for prediction. After these tokens are selected, the sequence is altered according to a specific modification strategy, which involves changing the selected tokens in one of three ways.

Token Selection and Modification Strategy in BERT's MLM

The pre-training pipeline for BERT's Masked Language Modeling (MLM) is a multi-step process. It begins with an input sequence, from which 15% of tokens are randomly selected. These chosen tokens are then altered: 80% are masked, 10% are replaced by random tokens, and 10% are left unchanged. This modified sequence is converted into embeddings and processed by a Transformer Encoder to produce contextualized hidden states. Finally, the model is trained using these hidden states to predict the original values of the altered tokens.

BERT's Masked Language Modeling Pre-training Pipeline

During the pre-training of language models, performance can begin to decline after a certain point. This degradation is sometimes attributed to interference, where learning new information negatively impacts previously learned knowledge. To counteract this, a practical strategy is to implement early stopping, which involves halting the training process to prevent such interference and preserve the model's optimal performance.

Performance Degradation and Early Stopping in Pre-training

The Masked Language Modeling (MLM) framework offers significant flexibility for training encoder-decoder models. Different training objectives can be created by adjusting various parameters, such as the percentage of tokens that are masked and the maximum length of the text spans that are replaced by a mask token. This adaptability allows the training to range from a BERT-style objective with partial masking to a full language modeling task where the entire sequence is generated.

Flexibility of Masked Language Modeling for Encoder-Decoder Training

As proposed in the original paper by Devlin et al. (2019), the standard BERT model is a Transformer encoder pre-trained with a dual-task objective. This training process involves simultaneously learning from two tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). The total training loss is calculated as the sum of the individual losses from these two objectives.

Training Objective of the Standard BERT Model

During a self-supervised pre-training process, a model is given an input sequence where one word has been replaced by a special symbol, for example: 'The quick brown [MASK] jumps over the lazy dog.' The model's objective is to predict the original word, 'fox'. Which of the following is the direct input used by the final output layer to make this specific prediction?

The sentence 'The puppies are frolicking outside the house .' serves as a base or original sequence for illustrating text corruption techniques. From this sentence, specific tokens can be selected to be either replaced with a `[MASK]` token (masking) or removed entirely (deletion), creating a noisy version for model training.

Original Sequence for Masking and Deletion Examples

Arrange the following steps in the correct order to describe the process of pre-training an encoder model using a masked language modeling objective.

A research team is developing a language model intended to be an expert in writing legal contracts. They are using a pre-training method where the model learns to fill in missing words in existing legal documents. Based *only* on this training objective of reconstructing corrupted text, critically evaluate the potential strengths and weaknesses of this approach for the model's final goal of generating complete, coherent legal clauses from scratch.

Learn Before

Related