Padding in Sequence Batching

Analyzing Batch Processing Challenges

A language model inference engine receives a batch of two user requests to process simultaneously for improved efficiency. The first request is 'Summarize the main causes of the Industrial Revolution in five points,' and the second is 'Define photosynthesis.' What is the primary computational challenge that arises from combining these specific requests into a single batch?

The Challenge of Variable-Length Sequences in Batch Processing