Example of Target Output Format for Span-Based Denoising
In a span-based denoising task, the model is trained to generate a target sequence that pairs each sentinel token from the input with the original text it replaced. For an input like [CLS] The puppies are [X] outside [Y] ., the corresponding target output would be ⟨s⟩ [X] frolicking [Y] the house [Z]. This format explicitly maps the sentinel [X] to 'frolicking' and [Y] to 'the house', with [Z] often serving as an end-of-sequence marker.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Example of Target Output Format for Span-Based Denoising
Consider the following sentence where two spans of text have been selected for masking: 'The team celebrated after their [innovative new model] achieved [state-of-the-art results].' How would this sentence be represented if unique sentinel tokens are used to replace each complete masked span?
Match each original sentence, with text to be masked enclosed in brackets, to its correct representation using unique sentinel tokens.
Applying Sentinel Tokens to Masked Spans
Learn After
A model is being trained on a denoising task where corrupted text spans are replaced by unique sentinel tokens. If the model receives the input
The [X] fox jumps over [Y] dog., what should the corresponding target output be to correctly associate[X]with 'quick brown' and[Y]with 'the lazy'?Constructing Target Output for Span-Based Denoising
A language model is being trained on a span-based denoising task where corrupted text spans are replaced by unique sentinel tokens. Match each corrupted input sentence on the left with its correctly formatted target output on the right.