Within-Example Parallelization Bottleneck in Recurrent Models
Due to their inherently sequential computation, recurrent models cannot parallelize processing across sequence positions within an individual training example. While parallel processing across distinct examples via batching can partially compensate at shorter lengths, batching becomes severely constrained by device memory limits when dealing with longer sequences, rendering the sequential bottleneck especially critical.
0
1
Tags
Prep Sessions
Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Ch.1 Transformer Architecture Fundamentals - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Sequential Computation Constraints in Recurrent Networks - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Learn After
Why does batching across distinct examples fail to fully resolve the parallelization bottleneck in recurrent models when sequence lengths increase?
What specific characteristic of recurrent models prevents them from parallelizing computation across sequence positions within an individual training example?
Contrast the parallelization capabilities of recurrent models at the within-example level versus the across-example level, and explain how sequence length influences the trade-off between them.