Learn Before
Prefix Cache Reuse Scenario
A system has pre-computed and stored the Key-Value (KV) cache state for the prefix ('Once', 'upon', 'a', 'time'). Now, a new input sequence ('Once', 'upon', 'a', 'star') needs to be processed. Based on the principle of how prefix cache states are generated, which pre-computed cache state can be reused to accelerate the processing of the new sequence, and why?
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
An auto-regressive model is processing the input sequence of tokens:
['The', 'cat', 'sat']. When the model uses the prefix['The', 'cat']to generate the next token,'sat', what is the content of the corresponding Key-Value (KV) cache state that is created at this step?An auto-regressive model is generating a series of Key-Value (KV) cache states for the input sequence of tokens:
['The', 'quick', 'brown']. Arrange the following events in the correct chronological order in which they occur during this process.Prefix Cache Reuse Scenario