Compression of Key-Value Pairs for Compressive Memory

FIFO Update of Compressive Memory

A long-context language model utilizes two distinct memory systems to manage information over time: a primary, fixed-size memory that holds recent, detailed information, and a secondary, compressed memory for older information. The primary memory operates by discarding its oldest entries to accommodate new data. Given this mechanism, what is the most direct source of information for updating the secondary, compressed memory?

A language model is designed with a two-tiered memory system to handle long documents. It has a fixed-size 'short-term memory' for recent, detailed information and a 'long-term memory' for older, summarized information. When a new segment of text is processed, arrange the following events in the correct chronological order to show how information flows between these two memory systems.

Relationship Between Memory Tiers in a Language Model