LLM Inference System Performance Diagnosis

An LLM inference system is reconfigured to handle long input sequences. Instead of processing the entire sequence in one large, parallel operation, it is broken down into smaller segments that are processed sequentially. This allows shorter, high-priority tasks to be interleaved. What is the most direct consequence of this change for the system's task scheduler?

Scheduling Overhead in LLM Inference