Learn Before
Request-Level Scheduling in LLM Inference
Request-level scheduling is a basic strategy for managing tasks in LLM inference. Under this approach, the scheduler groups requests into a complete batch and sends it to the inference engine. Once execution begins, the batch cannot be interrupted or modified. The scheduler is forced to wait until the entire batch finishes processing before it can dispatch the next one.
0
1
Tags
Foundations of Large Language Models
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Scheduler-Driven Batch Adjustments Between Iterations in Continuous Batching
An LLM inference system is receiving a high volume of requests. In its queue are several short, low-priority requests and one long, high-priority request. To maximize overall system efficiency, what is the most probable action the scheduler component will take?
Diagnosing LLM Inference System Performance Issues
Analyzing Scheduler Trade-offs in LLM Inference
Request-Level Scheduling in LLM Inference
Iteration-Based Scheduling in LLM Inference