A machine learning team observes that the initial processing of a user's entire input sequence is the slowest part of their language model's inference pipeline. This step involves a single, large computational pass where attention is calculated for all input tokens simultaneously. To reduce this latency, they can only afford one of the following hardware upgrades. Which upgrade would most effectively speed up this specific initial processing step?

Performance Bottleneck Analysis in LLM Inference

The prefilling phase of a large language model is considered a memory-bound process because the parallel computation of self-attention across the entire input sequence necessitates frequent and rapid data transfers to and from the processing unit's memory.