FIFO Update of Compressive Memory
The compressive memory () in the Compressive Transformer operates as a First-In, First-Out (FIFO) queue. When updating this memory, the newly compressed key-value pairs are appended to the tail of the queue. Simultaneously, to maintain its capacity, the first key-value pairs are dropped from the front of the queue.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Compression of Key-Value Pairs for Compressive Memory
FIFO Update of Compressive Memory
A long-context language model utilizes two distinct memory systems to manage information over time: a primary, fixed-size memory that holds recent, detailed information, and a secondary, compressed memory for older information. The primary memory operates by discarding its oldest entries to accommodate new data. Given this mechanism, what is the most direct source of information for updating the secondary, compressed memory?
A language model is designed with a two-tiered memory system to handle long documents. It has a fixed-size 'short-term memory' for recent, detailed information and a 'long-term memory' for older, summarized information. When a new segment of text is processed, arrange the following events in the correct chronological order to show how information flows between these two memory systems.
Relationship Between Memory Tiers in a Language Model
Learn After
Formula for FIFO Update of Compressive Memory
A compressive memory component is maintained as a fixed-size queue that can hold a maximum of 4 items. The current state of the memory is
[Item_1, Item_2, Item_3, Item_4], whereItem_1is the oldest item. If a new compressed item,Item_5, is added to this memory, what will be its resulting state?Compressive Memory Update Anomaly
A compressive memory component is implemented as a queue with a maximum capacity of 3 items. Initially, it contains
[C1, C2, C3], whereC1is the oldest item. Subsequently, two new compressed items,C4and thenC5, are added to the memory one after the other. Arrange the items that are removed from the memory in the correct chronological order of their removal.