Google

The T5 model utilizes a unified text-to-text format for all training samples, structured as $$\text{Source Text} \to \text{Target Text}$$. Within this structure, the source text provides the system with both a task description or instruction and the specific input data. The target text represents the expected response to the input task. By maintaining this consistent format, the model is able to frame and process many different problems as the same text-to-text task.

T5 Sample Format

A training sample for a machine translation task, such as Chinese to English, illustrates the T5 format. The sample includes a task-specific prefix, the input text, and the target translation. Special tokens structure the data for the model, for example: `[CLS] Translate from Chinese to English: 你好！ → ⟨s⟩ Hello!`. In this format, `[CLS]` serves as the start symbol for the source text (encoder input), while `<s>` is the start symbol for the target text (decoder input).

Example of a T5 Machine Translation Training Sample with Special Tokens

The T5 text-to-text framework can be applied to question-answering tasks by formatting both the input and the target output as text strings. For instance, a question is presented with a prefix like 'Answer:', and the model is trained to generate the corresponding answer. A complete example of this format is: `[CLS] Answer: when was Albert Einstein born? → ⟨s⟩ He was born on March 14, 1879.`

Example of a T5 Question-Answering Sample

The T5 text-to-text framework can be adapted for text simplification tasks by formatting both the input and target output as text strings. This is achieved by providing the model with an input string that includes a task-specific prefix, such as 'Simplify:', followed by the complex text. The model is then trained to generate the simplified version. A complete example of this format is: `[CLS] Simplify: the professor, who has published numerous papers in his field, will be giving a lecture on the topic next week. → ⟨s⟩ The experienced professor will give a lecture next week.`

Example of a T5 Simplification Task Sample

While it is possible to use a single, identical start symbol for both encoder and decoder input sequences, a common practice is to employ distinct symbols. This approach helps to clearly differentiate between the source sequence fed into the encoder and the target sequence processed by the decoder.

Differentiating Encoder and Decoder Sequences with Start Symbols

The T5 text-to-text framework is not confined to a single application, such as machine translation. Its 'Source Text → Target Text' structure can be generalized to represent a wide array of other natural language processing tasks, which are specified using task-specific prefixes.

Versatility of the T5 Text-to-Text Format

The variable $c_{gold}$ is used to denote the correct, or ground-truth, label for a given training sample.

Definition of c_gold

The input embedding for a model, denoted as $e$, is calculated by summing three component vectors: $x$, $e_{pos}$, and $e_{seg}$. This operation is represented by the formula: $e = x + e_{pos} + e_{seg}$.

Formula for Input Embedding Composition

A researcher wants to train a model to perform a new task: converting a sentence from passive voice to active voice. Given the passive input sentence 'The cake was eaten by the dog' and the desired active output 'The dog ate the cake', which of the following training samples is correctly structured according to the unified, prefix-based text-to-text format?

A machine learning engineer is preparing a dataset to train a model on a sentiment analysis task. They create the following training sample: `"The film was a masterpiece. → positive"`. Based on the principles of a unified, prefix-based text-to-text framework, identify the primary weakness of this sample format and explain how you would correct it to improve the model's ability to handle multiple, distinct tasks.

Critiquing a Text-to-Text Training Sample

A single text-to-text model is being trained on a dataset containing samples for four different tasks. Each sample's input begins with a prefix that instructs the model on what to do. Match each input sample (Source Text) with the most likely task it is intended for.

Learn Before

Related