A language model processes a long document by dividing it into 10 equal, non-overlapping segments. To maintain context, the model's attention mechanism at any point can access information from the segment it is currently processing as well as the single segment that came immediately before it. If the model is currently processing Segment 6, which segments' information is available to its attention mechanism?

Analyzing Context Limitations in a Recurrent Model

Analyzing Memory Trade-offs in Segment-Level Recurrence

Compressive Transformer Memory Architecture