Essay

Memory Overhead in Dynamic Sequence Generation

An LLM inference system is generating a long, complex story where the exact length of each new sentence is unknown beforehand. Compare the memory management operations and associated overhead for the key-value cache in two scenarios: 1) a system using a traditional approach that requires a single, contiguous memory block for the entire cache, and 2) a system using an approach that partitions the cache into smaller, non-contiguous blocks. In your analysis, explain which system is better suited for this task and why.

0

1

Updated 2025-10-06

Contributors are:

Who are from:

Tags

Ch.5 Inference - Foundations of Large Language Models

Foundations of Large Language Models

Foundations of Large Language Models Course

Computing Sciences

Analysis in Bloom's Taxonomy

Cognitive Psychology

Psychology

Social Science

Empirical Science

Science