Google

To illustrate the use of sentinel tokens for masked spans, consider a training example where the consecutive words 'frolicking' and 'the house' are masked. The corrupted input sequence is represented with placeholder slots:
`[CLS] The puppies are [X] outside [Y] .`
The training task is to generate a target output sequence that fills these slots with the correct tokens, which can be expressed as:
`<s> [X] frolicking [Y] the house [Z]`
Here, the unique sentinel tokens `[X]` and `[Y]` replace the masked spans in the input, and the target sequence uses these same tokens to provide the missing text, concluding with a final sentinel `[Z]`.

Example of Representing Masked Spans with Sentinel Tokens

In a span-based denoising task, the model is trained to generate a target sequence that pairs each sentinel token from the input with the original text it replaced. For an input like `[CLS] The puppies are [X] outside [Y] .`, the corresponding target output would be `⟨s⟩ [X] frolicking [Y] the house [Z]`. This format explicitly maps the sentinel `[X]` to 'frolicking' and `[Y]` to 'the house', with `[Z]` often serving as an end-of-sequence marker.

Example of Target Output Format for Span-Based Denoising

Match each original sentence, with text to be masked enclosed in brackets, to its correct representation using unique sentinel tokens.

A sentence has two sections identified for masking: 'The delivery driver dropped off the [large, heavy package] at the [front door].' Transform this sentence by replacing each of the two bracketed spans with its own unique sentinel token (e.g., `[X]`, `[Y]`).

Applying Sentinel Tokens to Masked Spans

Consider the following sentence where two spans of text have been selected for masking: 'The team celebrated after their [innovative new model] achieved [state-of-the-art results].' How would this sentence be represented if unique sentinel tokens are used to replace each complete masked span?

Learn Before

Related