Formula for FIFO Memory Update in Compressive Transformer

Compressive Memory Update in Compressive Transformer

A model's local memory component has a fixed capacity of 4 segments and operates on a First-In, First-Out (FIFO) basis. The memory currently holds the segments [Seg1, Seg2, Seg3, Seg4], where Seg1 is the oldest segment. If a new segment, Seg5, is processed, what will be the resulting state of the memory after the update?

The local memory in a specific transformer model is updated using a First-In, First-Out (FIFO) process to maintain a constant size. Put the two main steps of this update process in the correct order after a new segment of data arrives.

Debugging a Transformer's Memory Behavior