Example of a Request Completing in Continuous Batching (Iteration 5)

An LLM inference system is actively generating tokens for two separate user requests that are already in progress. A third user submits a new request to the system. To maximize overall throughput by overlapping different types of computation, what actions will the system perform in the next single computational step?

LLM Inference Scheduling Decision

Efficiency of Concurrent LLM Operations