An engineer is monitoring a text generation inference server that groups incoming requests into batches. They observe that while the time-to-completion for any single request within a running batch is very fast, the server's overall throughput (requests processed per hour) is low, with significant periods of hardware idleness. What is the most likely cause of this performance profile?

Analysis of Batch Processing Trade-offs

Evaluating an LLM Inference Strategy for a Real-Time Chatbot