State of Batched Requests After First Generation Step
An LLM inference system is processing two user prompts, 'Prompt A' and 'Prompt B', together in a batch. The system has just completed the initial processing of the input text for both prompts and has now performed exactly one subsequent generation step. Describe the state of both 'Prompt A' and 'Prompt B' at this moment.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Example of the Second Decoding Step in Continuous Batching (Iteration 3)
An inference engine is processing user requests. It has just completed the initial processing for two separate requests, 'Request A' and 'Request B', loading them into memory. Both requests are now ready for the next stage of generation. What is the most likely immediate next action the engine will take to operate efficiently, and what will be its result?
State of Batched Requests After First Generation Step
Inference Engine State Analysis