Given the scenario below, what action should the scheduler take regarding the new request, and why?

Google

In continuous batching, the scheduler can dynamically incorporate new user requests into the active batch between processing iterations. This action is conditional on the inference engine possessing sufficient available capacity to manage the additional workload. This allows the system to maintain high utilization by promptly integrating incoming tasks without waiting for the entire current batch to complete.

Adding New Requests in Continuous Batching

In continuous batching, if new user requests arrive when the inference engine is operating at full capacity, the scheduler does not add them to the active batch immediately. Instead, these requests are placed in a queue and must wait until resources are freed up, for instance, after an existing sequence in the batch completes its generation.

Queueing Requests in Continuous Batching

Dynamic Request Scheduling Scenario

An inference engine using a continuous batching strategy is actively processing a set of user requests. In the brief interval between two processing iterations, the scheduler successfully incorporates a newly arrived request into the active batch. What is the most critical condition that must have been met for the scheduler to make this decision?

In a system using continuous batching, a new user request that arrives while an existing batch is being processed must wait until all requests in that current batch are fully completed before it can be considered for processing.

Learn Before

Related