These objectives restore the processed text to its uncorrupted state by calculating loss over only the noised parts of the input sentence.

University of Michigan - Ann Arbor

1. Standard Language Model(SLM)
2. Corrupted Text Reconstruction(CTR)
3. Full Text Reconstruction(FTR)

Types of Training Objectives of Pre-trained LM

https://arxiv.org/pdf/2107.13586.pdf
It's one amazing survey paper proposing existing prompt-tuning methods.

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing


Corrupted Text Reconstruction(CTR)

These objectives reconstruct the text by calculating the loss over the entirety of the input texts whether it has been noised or not.

Full Text Reconstruction (FTR)

Standard Language Modeling is a pre-training objective that trains a model to optimize the probability of a text sequence, $P(x)$, from a given corpus. This is typically achieved through an auto-regressive generation procedure where the model predicts each subsequent token based on its preceding context.

Learn Before

Related