Learn Before
Continuous Batching Scheduler Behavior
Analyze the following scenario and describe the state of the active batch and the waiting queue immediately after the final event occurs. Provide a step-by-step explanation for your answer.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
An inference engine using a continuous batching scheduler is operating at maximum capacity, meaning it cannot immediately process any more sequences. When new user requests arrive under these conditions, they are placed in a waiting queue. What is the primary trade-off the system is making by implementing this queueing mechanism?
An inference engine using a continuous batching strategy is currently processing a set of text generation requests that fully utilizes its processing capacity. At this point, a new, additional request arrives. What is the most likely immediate action the system's scheduler will take regarding this new request?
Continuous Batching Scheduler Behavior