A development team is using a large language model for two different tasks. Task A requires generating a response to a user's query as quickly as possible to maintain a conversational flow. Task B involves processing a large collection of documents where the total time to complete all documents is the main concern, but the time for any single document is less critical. To achieve the fastest possible response time for an individual query in Task A, which processing approach should be used and why?
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Foundations of Large Language Models Course
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A development team is using a large language model for two different tasks. Task A requires generating a response to a user's query as quickly as possible to maintain a conversational flow. Task B involves processing a large collection of documents where the total time to complete all documents is the main concern, but the time for any single document is less critical. To achieve the fastest possible response time for an individual query in Task A, which processing approach should be used and why?
Latency in Batched vs. Single Sequence Processing
When a system processes a single input sequence at a time, the latency for that request is minimized because there is no added delay from waiting for other sequences in a batch to complete their generation.