An inference engine is processing user requests. It has just completed the initial processing for two separate requests, 'Request A' and 'Request B', loading them into memory. Both requests are now ready for the next stage of generation. What is the most likely immediate next action the engine will take to operate efficiently, and what will be its result?
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Example of the Second Decoding Step in Continuous Batching (Iteration 3)
An inference engine is processing user requests. It has just completed the initial processing for two separate requests, 'Request A' and 'Request B', loading them into memory. Both requests are now ready for the next stage of generation. What is the most likely immediate next action the engine will take to operate efficiently, and what will be its result?
State of Batched Requests After First Generation Step
Inference Engine State Analysis