Impact of Batch Size on the Throughput-Latency Trade-off

An engineering team is optimizing a system that serves a large language model to multiple users. To maximize the number of requests processed per hour, they decide to group incoming requests into large batches before sending them to the hardware for processing. This approach significantly increases the system's overall processing capacity. For which of the following applications would this optimization strategy be most detrimental to the user experience?

Optimizing LLM Serving for Different Applications

The Core Trade-off in LLM Serving