Google

To illustrate the use of sentinel tokens for masked spans, consider a training example where the consecutive words 'frolicking' and 'the house' are masked. The corrupted input sequence is represented with placeholder slots:
`[CLS] The puppies are [X] outside [Y] .`
The training task is to generate a target output sequence that fills these slots with the correct tokens, which can be expressed as:
`<s> [X] frolicking [Y] the house [Z]`
Here, the unique sentinel tokens `[X]` and `[Y]` replace the masked spans in the input, and the target sequence uses these same tokens to provide the missing text, concluding with a final sentinel `[Z]`.

Example of Representing Masked Spans with Sentinel Tokens

In a span-based denoising task, the model is trained to generate a target sequence that pairs each sentinel token from the input with the original text it replaced. For an input like `[CLS] The puppies are [X] outside [Y] .`, the corresponding target output would be `⟨s⟩ [X] frolicking [Y] the house [Z]`. This format explicitly maps the sentinel `[X]` to 'frolicking' and `[Y]` to 'the house', with `[Z]` often serving as an end-of-sequence marker.

Example of Target Output Format for Span-Based Denoising

Consider the following sentence where two spans of text have been selected for masking: 'The team celebrated after their [innovative new model] achieved [state-of-the-art results].' How would this sentence be represented if unique sentinel tokens are used to replace each complete masked span?

Match each original sentence, with text to be masked enclosed in brackets, to its correct representation using unique sentinel tokens.

A sentence has two sections identified for masking: 'The delivery driver dropped off the [large, heavy package] at the [front door].' Transform this sentence by replacing each of the two bracketed spans with its own unique sentinel token (e.g., `[X]`, `[Y]`).

Learn Before

Related