Based on the following observation, what specific operation at the beginning of the text generation process is likely responsible for the initial, input-length-dependent delay, and what is its fundamental purpose?

Google

The inference process within the prefilling-decoding framework begins by establishing an initial context, denoted as 'x'. This input sequence serves as the foundation for the prefilling phase, where its representation is computed and stored in the KV cache.

Establishing the Initial Context for Inference

The prefilling phase is the initial stage of Transformer inference where the model processes the input sequence, denoted as `x`, to compute and populate the Key-Value (KV) cache. This stage is named 'prefilling' because its primary function is to prepare and store the key-value vector pairs for every token in the input prompt before the generative decoding process begins.

Prefilling Phase in Transformer Inference

A user provides the following sequence of words to a large language model: 'Write a short story about a robot who discovers music.' In the model's text generation process, what is the primary role of this initial sequence of words?

Diagnosing Inference Latency

A user submits a long document to a text-generation model with the instruction "Summarize this document." In the context of the model's inference process, explain the two primary functions of this initial document *before* any new words of the summary are generated.

Learn Before

Related