An LLM inference system is designed with two specialized hardware engines operating in a pipeline. Engine A processes the initial prompts for a batch of user requests to generate their internal state. This state is then passed to Engine B, which handles the step-by-step generation of the response tokens for that same batch. As soon as Engine A finishes with the first batch, it immediately begins processing the initial prompts for a second, new batch of requests while Engine B is still generating tokens for the first batch. What is the primary computational advantage of this two-engine architecture?

Optimizing LLM Inference Throughput

Example of Pipelined Prefilling and Decoding with Two Engines

Pipelined Engine Efficiency