Formula for Input Embedding Composition
The final input embedding for a sequence token, denoted as , is calculated by summing three distinct component vectors: the token embedding , the positional embedding , and the segment embedding . This composition is represented by the formula:
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
A researcher is training a sequence-processing model and observes that while it correctly identifies the meaning of individual words, it consistently fails on tasks where word order is crucial. For example, it treats 'dog bites man' and 'man bites dog' as having the same overall meaning. The researcher suspects an issue in how the initial input vectors are constructed for the model. What is the most probable cause of this issue?
Constructing an Input Vector for a Sequence Model
Calculating a Combined Input Vector
Learnable Absolute Positional Embeddings
Formula for Input Embedding Composition
Example of Input Embedding Composition for a Sentence Pair
A model processes a two-sentence input: 'The sky is blue. [SEP] Grass is green.'. To help the model distinguish between the two sentences, it uses a specific vector,
Vec_A, for the first sentence and another vector,Vec_B, for the second. How are these vectors assigned to the tokens in the combined input sequence?Debugging a Sentence-Pair Model
A model is given a two-sentence input: 'What is the capital of France? [SEP] Paris is the capital.'. The model uses one vector representation for the first sentence (let's call it
Vec_A) and a different one for the second sentence (Vec_B). For the tokenized sequence below, what is the correct sequence of these vector labels that would be assigned to each token?[CLS] What is the capital of France ? [SEP] Paris is the capital . [SEP]The correct sequence is: ____. (Use a comma and space to separate labels, e.g.,
Label1, Label2)Formula for Input Embedding Composition
Learn After
A researcher designs a language model where the final input representation for each word is created by summing a vector for the word's identity and a vector for the sentence it belongs to. However, they intentionally omit the vector that encodes the word's specific position in the sequence. What is the most likely deficiency this model will exhibit?
Calculating a Final Input Embedding
A common method for creating the final input representation for a token in a sequence involves summing three distinct vectors. Match each vector component to its specific function in this process.