Learn Before
Comparison of Prefilling and BERT Encoding
The prefilling phase is analogous to BERT's encoding process in that both convert an input sequence into contextualized token representations. However, they differ fundamentally in their operational mode: prefilling is unidirectional, consistent with standard autoregressive language modeling, whereas BERT employs a bidirectional approach to generate its representations.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Comparison of Prefilling and BERT Encoding
A machine learning engineer observes that the initial processing of a user's prompt in a large language model takes a significant amount of time, but subsequent token generation is much faster per token. Based on this observation, which statement best analyzes the primary function of this initial processing phase (prefilling)?
Objectives of Inference Phases
The main goal of the prefilling phase in a generative language model is to generate the first token of the model's response, while the computation of the input sequence's contextual representation is a secondary effect of this process.
Learn After
Consider two different methods for creating contextualized numerical representations of words in a sentence. Method 1 generates a representation for each word based only on the words that precede it. Method 2 generates a representation for each word based on all other words in the sentence, both preceding and succeeding it. Which statement accurately compares these two methods to the processes found in large-scale language models?
Directionality in Contextual Representations
The primary functional difference between the prefilling phase in an autoregressive model and the encoding process in a model like BERT is the specific mathematical operations used to create token representations; their approach to incorporating contextual information from the input sequence is fundamentally identical.