Comparison of Processing in Chunked vs. Standard Prefilling

A large language model is tasked with processing a very long input document. To prepare for generating a response, it computes the Key-Value cache for the entire document in a single, large forward pass before any new tokens are produced. What is the most significant computational challenge or trade-off inherent to this 'all-at-once' approach?

A user submits a prompt to a large language model that uses a conventional inference process. Arrange the following stages in the correct chronological order, from receiving the prompt to generating the first new word.

Inference Bottleneck on Memory-Constrained Devices