Inference Engine State Analysis
An inference engine is processing two user prompts simultaneously. The engine has just completed the initial processing phase where it has understood both prompts. It now executes a single, unified generation operation for both prompts together. Based on how batched processing works in this context, what is the most likely state of the generated output for each prompt immediately after this operation, and why?
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Example of the Second Decoding Step in Continuous Batching (Iteration 3)
An inference engine is processing user requests. It has just completed the initial processing for two separate requests, 'Request A' and 'Request B', loading them into memory. Both requests are now ready for the next stage of generation. What is the most likely immediate next action the engine will take to operate efficiently, and what will be its result?
State of Batched Requests After First Generation Step
Inference Engine State Analysis