Learn Before
Example of Sequence Packing for Translation
When preparing parallel text for sequence models, such as translation pairs, the source and target sentences can be packed into a single concatenated sequence using special tokens. For example, a Chinese sentence "鲸鱼 是 哺乳 动物 。" and its English translation "Whales are mammals ." can be packed as: [CLS] 鲸鱼 是 哺乳 动物 。 [SEP] Whales are mammals . [SEP]. In this structure, the [CLS] token marks the beginning of the sequence, while [SEP] tokens are used to separate the two languages and indicate the end of the entire sequence.
0
1
Tags
Foundations of Large Language Models
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
A language model is being prepared for a question-answering task. The model must process both the question and its corresponding answer as a single, combined sequence. If the question is 'What is the capital of France?' and the answer is 'Paris', how should these two sequences be formatted for the model using a special separator token to distinguish between them?
Diagnosing Model Training Issues from Data Formatting
Debugging Data Preprocessing for a Summarization Model
Example of Sequence Packing for Translation