Learn Before
Token-Level Representation of Input and Output Sequences for a Forward Pass
A forward pass in a language model involves processing concatenated token sequences representing both the input and the output. For instance, an input sequence denoted by tokens x0 x1 x2 x3 is combined with the generated output tokens, such as y1 and y2. The model first processes the input to predict y1, and subsequently processes the combined sequence of input and y1 to predict the next token, y2.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Mathematical Formulation of LLM Inference
Single-Round Prediction Problem
Token-Level Representation of Input and Output Sequences for a Forward Pass
Multi-Round Prediction Problem
Notation for Concatenated Token Sequences
A language model is given an input sequence of tokens representing the phrase 'The best way to learn a new skill is'. The model then calculates the likelihood for several possible completing sequences. Based on the formal objective of the text generation process, which of the following sequences should the model select to output?
Analyzing Model Output Selection
A language model is given an input context
x. It then evaluates two potential output sequences,y_1andy_2. The model's internal calculations determine thaty_1has a higher probability of occurring afterxthany_2. However, a human evaluator findsy_2to be more creative and detailed. According to the formal objective of the text generation process, what should the model do?
Learn After
A language model is given the input 'The cat sat on' and begins to generate the next part of the sentence. The model's first generated word is 'the'. To predict the second word of its output, what sequence of words does the model use as its new input for the next processing step?
A text-generating model is given the input sequence of tokens:
['The', 'quick', 'brown', 'fox']. It then generates the output sequence:['jumps', 'over', 'the']. Arrange the following processing steps in the correct chronological order that the model would take to produce this output.Debugging a Text Generation Process