Google

The choice of batch size creates a direct trade-off between system throughput and latency. Smaller batch sizes result in lower latency because fewer tokens are processed in one inference pass. This approach, however, leads to underutilized parallel computing resources, such as GPUs remaining idle, which diminishes overall system throughput. In contrast, larger batch sizes maximize throughput by fully engaging the hardware's parallelism, allowing GPUs to be occupied with large-scale matrix computations. This efficiency gain comes with higher latency, as results are only available after the entire batch, including the final token, has been processed.

Impact of Batch Size on the Throughput-Latency Trade-off

Analyze the two deployment scenarios described below. For each scenario, recommend whether to use a larger or smaller request batch size to optimize performance. Justify your recommendations by explaining the resulting trade-offs between overall processing efficiency and the time it takes to get a response for a single request.

Optimizing LLM Serving Configuration

An engineering team is deploying a large language model to power a real-time, interactive customer service chatbot. The top priority is ensuring that users experience minimal delay between sending a message and receiving a response. Which batch size strategy should the team implement to best achieve this goal?

An example illustrating the efficiency gains from batching compares processing four sequences with a batch size of one versus a batch size of four. When the batch size is one, each sequence is processed sequentially, with the system completing the prefilling and decoding for the first sequence before starting the second, and so on. In contrast, with a batch size of four, all four sequences are processed in parallel within a single computational pass. This parallel execution significantly increases throughput by making better use of the hardware's capacity, even though it requires padding shorter sequences to match the length of the longest one.

Example of Throughput Gain with Increased Batch Size

An illustrative case for understanding latency is processing a single input sequence. In this scenario, with a batch size of one, the result becomes available immediately after the generation is complete. There is no additional waiting time or computational overhead caused by other sequences, representing the lowest possible latency for a request.

Learn Before

Related