Example of a Denoising Autoencoder Task for Encoder-Decoder Models
This example illustrates the input and output for an encoder-decoder model undergoing denoising training. The encoder processes a corrupted input sequence where specific tokens have been replaced by a mask token, represented as: [CLS] The puppies are [MASK] outside [MASK] house . The decoder then utilizes the hidden representation from the encoder to predict and reconstruct the original, uncorrupted sequence, yielding: ⟨s⟩ The puppies are frolicking outside the house .
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Example of Denoising Task with Consecutive Token Masking
Span-Based Denoising as an Encoder-Decoder Training Objective
Input Corruption Methods for Denoising Autoencoder Training
Denoising Autoencoder Training Objective
Loss Calculation for Encoder-Decoder Denoising Tasks
Training Efficiency in Denoising Autoencoding
Flexibility of Masked Language Modeling for Encoder-Decoder Training
Example of a Denoising Autoencoder Task for Encoder-Decoder Models
BART Model's Use of Diverse Input Corruption Methods
An encoder-decoder model is being trained using the following example:
- Input to Encoder: "The scientist carefully [MASK] the solution into the beaker."
- Target Output for Decoder: "The scientist carefully poured the solution into the beaker."
Based on this training setup, what is the primary function of the decoder?
Evaluating a Model Training Objective
An encoder-decoder model is being trained with the objective of reconstructing a full, original sentence from an input version where several random words have been removed. What is the most critical function of the encoder's output in this specific training paradigm?
Corrupted Input for Encoder-Decoder Pre-training
Diagrammatic Example of an Encoder-Decoder Model Trained with a Denoising Autoencoding Objective
Learn After
A model is being trained to reconstruct an original text sequence from a corrupted version where some words have been replaced. If the model is given the corrupted input:
The quick brown [MASK] jumps over the [MASK] dog., which of the following would be the most appropriate target output for this training example?Evaluating a Model's Training Objective
Focus of the Denoising Loss Function