Google

An illustration of the initial step in continuous batching involves the scheduler receiving new requests, such as 'x1' and 'x2'. In the first iteration of the process, these requests are formed into an initial batch and sent to the inference engine to begin the prefilling phase.

Example of Initial Batch Creation in Continuous Batching

This diagram illustrates the second iteration in a continuous batching process, which follows the initial prefilling of requests x1 and x2. In this step, the scheduler directs the inference engine to perform a single decoding operation for the entire batch. This concurrently generates the first output token for both request x1 and request x2, demonstrating how the system transitions from the prefilling phase to the iterative decoding phase for a group of requests.

Example of the First Decoding Step in Continuous Batching (Iteration 2)

An inference server's scheduler receives two new, independent user requests at the same time. Assuming the system has the capacity to handle both, what is the most accurate description of the scheduler's immediate action and the primary goal of this initial processing step?

An LLM inference server employing a continuous batching strategy receives three new, independent user prompts at the same time. Describe the immediate action the system's scheduler will perform with these prompts and name the initial processing phase that this action initiates.

Initial Batch Formation

An inference scheduler receives two new, independent requests. Arrange the following events to accurately describe the initial processing step for these requests.

Learn Before

Related