Google

In continuous batching, an iteration represents a distinct step in computation, corresponding to either the full prefilling phase for a given input or a single token's decoding step. For instance, given an input sequence $$\mathbf{x}=x_0...x_m$$ and a target output sequence $$\mathbf{y}=y_1...y_n$$, processing requires a total of $$n+1$$ iterations. This includes one initial iteration to handle prefilling, followed by $$n$$ iterations to generate the output sequence, yielding one token per iteration.

Iteration in Continuous Batching

A large language model using a continuous batching inference system processes a single request. The input prompt consists of 150 tokens, and the model is configured to generate an output of 200 tokens. How many computational iterations are required to fully process this single request?

After an inference system has completed a total of 12 computational iterations since the requests arrived, what is the status of each request? Specifically, state whether each request is complete and, if not, how many output tokens have been generated for it. Justify your reasoning based on how iterations are defined for the input processing and output generation phases.

LLM Inference Request Processing

In a continuous batching system for large language model inference, every single token processed, whether from the input prompt or the generated output, constitutes one separate computational iteration.

Learn Before

Related