Google

This strategy, known as the disaggregation of prefilling and decoding, implements continuous batching by using two specialized hardware engines. A dedicated 'Engine 1' performs prefilling for a batch of requests. Once complete, the generated Key-Value (KV) cache is sent to a separate 'Engine 2' for decoding. The primary benefit of this pipeline is that Engine 1 can immediately start prefilling the next batch while Engine 2 is decoding the first. This overlapping of computations is key to improving computational efficiency and maximizing hardware utilization.

Disaggregation of Prefilling and Decoding using Pipelined Engines

Based on the provided scenario, propose a specific architectural change to the inference system that would address the described underutilization and idle time. Explain precisely how your proposed change would lead to better hardware utilization and increased user throughput.

Optimizing LLM Inference Throughput

This diagram illustrates a pipelined architecture for LLM inference that uses two separate engines to improve efficiency. Engine 1 is dedicated to the prefilling phase, processing an initial batch of requests (e.g., sequences 1-4). Once the prefilling is complete, the resulting Key-Value (KV) cache is transferred to Engine 2, which then handles the decoding phase for that batch. The key advantage of this disaggregated approach is that Engine 1 can immediately begin prefilling a new batch of requests (e.g., sequences 5-6) while Engine 2 is concurrently decoding the first batch, thus overlapping computations and maximizing hardware utilization.

Example of Pipelined Prefilling and Decoding with Two Engines

An LLM inference system uses two separate engines in a pipeline: Engine 1 for processing initial prompts (prefilling) and Engine 2 for generating subsequent tokens (decoding). When a continuous stream of request batches arrives, explain precisely *when* Engine 1 can start processing a new batch (e.g., Batch B) in relation to the processing of the previous batch (Batch A). Why is this timing crucial for maximizing hardware utilization?

Pipelined Engine Efficiency

An LLM inference system is designed with two specialized hardware engines operating in a pipeline. Engine A processes the initial prompts for a batch of user requests to generate their internal state. This state is then passed to Engine B, which handles the step-by-step generation of the response tokens for that same batch. As soon as Engine A finishes with the first batch, it immediately begins processing the initial prompts for a second, new batch of requests while Engine B is still generating

Learn Before

Related