Google

The denoising autoencoding objective is utilized to train encoder-decoder models by requiring them to reconstruct an original, uncorrupted sequence from a corrupted input. Operating similarly to a denoising autoencoder, the encoder processes the corrupted input—such as a sequence with masked tokens—and transforms it into a hidden representation. The decoder then uses this hidden representation to predict the original text. By learning to map a corrupted sequence to its uncorrupted counterpart, the model concurrently develops two key skills: the encoder gains the ability to comprehend the input context, while the decoder acquires the capability to generate coherent text.

Training Encoder-Decoder Models with a Denoising Autoencoding Objective

An example of a denoising autoencoder task involving consecutive token masking is training a model to reconstruct a full sentence from a corrupted input. The model receives an input where adjacent tokens have been replaced by `[MASK]`, and its objective is to generate the original, complete sequence. For instance, the input `[CLS] The puppies are [MASK] outside [MASK] [MASK] .` would be used to train the model to produce the target output: `⟨s⟩ The puppies are frolicking outside the house .`.

Example of Denoising Task with Consecutive Token Masking

In a span-based denoising task for an encoder-decoder model, the training objective is to reconstruct only the original text from masked spans. The model's encoder processes an input sequence where one or more spans of text have been replaced by unique mask or sentinel tokens. The decoder is then trained to generate a sequence containing these sentinel tokens paired with the original text they replaced, effectively learning to 'fill in the blanks'. A loss function is computed by comparing the generated text with the ground-truth masked spans.

Span-Based Denoising as an Encoder-Decoder Training Objective

When training encoder-decoder models with a denoising autoencoding objective, various methods can be used to corrupt the input data. This process is crucial for training the model to reconstruct the original input. Besides the common technique of masking tokens, other corruption strategies include altering tokens to different ones or reordering them within the sequence.

Input Corruption Methods for Denoising Autoencoder Training

The training objective of a denoising autoencoder is to identify the optimal parameters for the encoder ($$\theta$$) and decoder ($$\omega$$) to minimize reconstruction error. During training, a corrupted input $$\mathbf{x}_{\mathrm{noise}}$$ is generated by adding noise to the original input $$\mathbf{x}$$. The model processes this noisy input, and the loss function—frequently chosen as cross-entropy loss—measures how effectively the decoder recovers the original $$\mathbf{x}$$. The objective to find the optimal parameters, $$\hat{\theta}$$ and $$\hat{\omega}$$, is mathematically defined as:

$$(\hat{\theta},\hat{\omega}) = \arg\min_{\theta,\omega} \mathrm{Loss}(\mathrm{Model}_{\theta,\omega}(\mathbf{x}_{\mathrm{noise}}),\mathbf{x})$$

Denoising Autoencoder Training Objective

When training an encoder-decoder model on a denoising objective, the loss is calculated across the entire output sequence. The decoder generates the target sequence one token at a time. At each generation step, a loss function, typically cross-entropy, measures the discrepancy between the model's predicted probability distribution for the next token and the actual ground-truth token. The total loss for the training example is then computed by summing or averaging these individual token-level losses over the full length of the target sequence.

Loss Calculation for Encoder-Decoder Denoising Tasks

In denoising autoencoding, corrupted text can be represented using placeholder slots for the masked or deleted tokens. The model is then trained to fill these slots with the original tokens by leveraging the surrounding context. A key benefit of this method is that using placeholders can result in shorter input sequences, which improves the computational efficiency of the training process.

Training Efficiency in Denoising Autoencoding

The Masked Language Modeling (MLM) framework offers significant flexibility for training encoder-decoder models. Different training objectives can be created by adjusting various parameters, such as the percentage of tokens that are masked and the maximum length of the text spans that are replaced by a mask token. This adaptability allows the training to range from a BERT-style objective with partial masking to a full language modeling task where the entire sequence is generated.

Flexibility of Masked Language Modeling for Encoder-Decoder Training

This example illustrates the input and output for an encoder-decoder model undergoing denoising training. The encoder processes a corrupted input sequence where specific tokens have been replaced by a mask token, represented as: `[CLS] The puppies are [MASK] outside [MASK] house .` The decoder then utilizes the hidden representation from the encoder to predict and reconstruct the original, uncorrupted sequence, yielding: `⟨s⟩ The puppies are frolicking outside the house .`

Example of a Denoising Autoencoder Task for Encoder-Decoder Models

After defining the model architecture and training objective for a denoising autoencoder, a key remaining step is to specify how the input data is corrupted. The BART model, developed by Lewis et al. (2020), exemplifies this by employing several different methods for corrupting the input sequence during its pre-training phase.

BART Model's Use of Diverse Input Corruption Methods

An encoder-decoder model is being trained using the following example:

- **Input to Encoder:** "The scientist carefully [MASK] the solution into the beaker."
- **Target Output for Decoder:** "The scientist carefully poured the solution into the beaker."

Based on this training setup, what is the primary function of the decoder?

A machine learning engineer sets up a training process for an encoder-decoder model where the encoder receives a text sequence with several words replaced by a `[MASK]` token. The model's objective is to have the decoder output the exact same sequence it received as input, including the `[MASK]` tokens. Critically evaluate this training objective. Would this approach be effective for teaching the model to generate coherent, uncorrupted text? Justify your reasoning.

Evaluating a Model Training Objective

An encoder-decoder model is being trained with the objective of reconstructing a full, original sentence from an input version where several random words have been removed. What is the most critical function of the encoder's output in this specific training paradigm?

When pre-training an encoder-decoder model using either BERT-style or denoising autoencoding methods, the initial step involves processing data through the encoder. The input provided to the encoder is a corrupted token sequence where some specific tokens have been intentionally masked and replaced with a special placeholder, such as `[MASK]` (or `[M]` for short).

Corrupted Input for Encoder-Decoder Pre-training

This diagram provides an example of training an encoder-decoder model using a denoising autoencoding objective. The process involves several key steps: 1. **Corrupted Input to Encoder:** The encoder receives a corrupted version of a sentence where some tokens are masked, for instance, `[CLS] The puppies are [MASK] outside [MASK] house .`. 2. **Sequence Reconstruction by Decoder:** The encoder generates a hidden state representation of the input, which is then passed to the decoder. The decoder's task is to reconstruct the original, uncorrupted sentence, `⟨s⟩ The puppies are frolicking outside the house .`, in an autoregressive manner. 3. **Sequence-Level Loss Calculation:** To train the model, a loss is calculated over the entire output sequence by accumulating the losses of all tokens, as in standard language modeling. This involves comparing the decoder's generated output with the ground-truth sequence, and the resulting error signal is used to update the model's parameters.

Learn Before

Related