A team is optimizing a text-generation model where the computational cost is dominated by the self-attention mechanism during autoregressive decoding. They need to decide between two potential upgrades:

1. Upgrade A: Doubling the number of layers in the model while keeping the maximum sequence length the same.
2. Upgrade B: Doubling the maximum sequence length the model can handle while keeping the number of layers the same.

Assuming the model generates a sequence that fills its m

Derivation of Quadratic Complexity in Autoregressive Attention

Performance Bottleneck in a Generative Model

Vector Products per Self-Attention Step