Optimizing LLM Serving Configuration

An engineering team is deploying a large language model to power a real-time, interactive customer service chatbot. The top priority is ensuring that users experience minimal delay between sending a message and receiving a response. Which batch size strategy should the team implement to best achieve this goal?

Example of Throughput Gain with Increased Batch Size

Example of Minimal Latency with a Single Sequence

Match each performance characteristic of a language model serving system with the batch size strategy that is its primary cause.