Example of Input Embedding Composition for a Sentence Pair
To demonstrate the composition of embeddings for a sentence pair (e.g., 'It is raining.' and 'I need an umbrella.'), each token receives three vectors: a token embedding (), a positional embedding (), and a segment embedding (). The segment embedding assigns to tokens of the first sentence and to tokens of the second. The structure is as follows:
| Token | [CLS] | It | is | raining | . | [SEP] | I | need | an | umbrella | . | [SEP] |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Example of Input Embedding Composition for a Sentence Pair
A model processes a two-sentence input: 'The sky is blue. [SEP] Grass is green.'. To help the model distinguish between the two sentences, it uses a specific vector,
Vec_A, for the first sentence and another vector,Vec_B, for the second. How are these vectors assigned to the tokens in the combined input sequence?Debugging a Sentence-Pair Model
A model is given a two-sentence input: 'What is the capital of France? [SEP] Paris is the capital.'. The model uses one vector representation for the first sentence (let's call it
Vec_A) and a different one for the second sentence (Vec_B). For the tokenized sequence below, what is the correct sequence of these vector labels that would be assigned to each token?[CLS] What is the capital of France ? [SEP] Paris is the capital . [SEP]The correct sequence is: ____. (Use a comma and space to separate labels, e.g.,
Label1, Label2)Example of Next Sentence Prediction (NSP) Input Formatting
Example of Input Embedding Composition for a Sentence Pair
A language model is designed to process pairs of sentences to understand their relationship. Given Sentence A: 'The team won the championship.' and Sentence B: 'The city celebrated all night.', which of the following options correctly structures these two sentences into a single input sequence for the model?
A developer is preparing input for a language model designed to analyze the relationship between two sentences. The first sentence is 'The sky is blue.' and the second is 'Grass is green.'. They construct the following input sequence:
[CLS] The sky is blue. [SEP] Grass is green.Which statement best analyzes the error in this input format?Analyzing Sentence-Pair Input Structure
A researcher is debugging a language model where the input representation for each token is created by summing three distinct vectors: one for the token's identity, one for its position in the sequence, and one for the sentence segment it belongs to. The researcher observes that the model treats the sentences 'The scientist observed the star' and 'The star observed the scientist' as having identical meanings. Which of the three component vectors is most likely being calculated incorrectly or omitted, causing this specific error?
In a language model that uses separate vectors for token identity, position, and sentence membership, the final input vector for a token is created by concatenating these three component vectors end-to-end.
Debugging Sentence Pair Representations
Segment Embedding
Example of Input Embedding Composition for a Sentence Pair
Learn After
Consider the following input prepared for a language model that processes sentence pairs:
[CLS] The sky is blue . [SEP] The grass is green . [SEP]The final vector representation for each token in this sequence is created by summing three distinct vectors: one for the token's identity, one for its position in the sequence, and one to indicate which sentence it belongs to (the first or the second).
Which of the following statements correctly analyzes the final vector representations for the two instances of the token 'The' in the sequence?
Deconstructing an Input Vector
Debugging an Input Embedding