An inference engine using a continuous batching scheduler is operating at maximum capacity, meaning it cannot immediately process any more sequences. When new user requests arrive under these conditions, they are placed in a waiting queue. What is the primary trade-off the system is making by implementing this queueing mechanism?

An inference engine using a continuous batching strategy is currently processing a set of text generation requests that fully utilizes its processing capacity. At this point, a new, additional request arrives. What is the most likely immediate action the system's scheduler will take regarding this new request?

Continuous Batching Scheduler Behavior