Google

In a span-based denoising task, the model is trained to generate a target sequence that pairs each sentinel token from the input with the original text it replaced. For an input like `[CLS] The puppies are [X] outside [Y] .`, the corresponding target output would be `⟨s⟩ [X] frolicking [Y] the house [Z]`. This format explicitly maps the sentinel `[X]` to 'frolicking' and `[Y]` to 'the house', with `[Z]` often serving as an end-of-sequence marker.

Example of Target Output Format for Span-Based Denoising

A large language model is being trained with a span-based denoising objective. Given the corrupted input text `The cat [X] on the [Y] mat.` and the knowledge that `[X]` replaced the text 'sat quietly' and `[Y]` replaced the text 'warm', construct the complete and correctly formatted target output sequence the model should be trained to generate. Assume the sequence should start with `<s>` and end with `[Z]`.

Constructing Target Output for Span-Based Denoising

A language model is being trained on a span-based denoising task where corrupted text spans are replaced by unique sentinel tokens. Match each corrupted input sentence on the left with its correctly formatted target output on the right.

A model is being trained on a denoising task where corrupted text spans are replaced by unique sentinel tokens. If the model receives the input `The [X] fox jumps over [Y] dog.`, what should the corresponding target output be to correctly associate `[X]` with 'quick brown' and `[Y]` with 'the lazy'?

Learn Before

Related