Mask, Replace, Delete, Permute, Rotate, Concatenation

Noising Functions

These objectives reconstruct the text by calculating the loss over the entirety of the input texts whether it has been noised or not.

University of Michigan - Ann Arbor

1. Standard Language Model(SLM)
2. Corrupted Text Reconstruction(CTR)
3. Full Text Reconstruction(FTR)

Types of Training Objectives of Pre-trained LM

https://arxiv.org/pdf/2107.13586.pdf
It's one amazing survey paper proposing existing prompt-tuning methods.

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing


These objectives restore the processed text to its uncorrupted state by calculating loss over only the noised parts of the input sentence.

Corrupted Text Reconstruction(CTR)

Full Text Reconstruction (FTR)

Standard Language Modeling is a pre-training objective that trains a model to optimize the probability of a text sequence, $P(x)$, from a given corpus. This is typically achieved through an auto-regressive generation procedure where the model predicts each subsequent token based on its preceding context.

Learn Before

Related

Learn After