Two-Model Architecture of Speculative Decoding

Speculative Decoding Algorithm

Evaluating an Inference Optimization Technique

A team is implementing an inference optimization technique where a small, fast model proposes a sequence of several tokens, and a large, accurate model then validates this entire sequence in a single step. What is the most critical factor for this technique to achieve a significant speedup compared to generating tokens one by one with the large model?

A development team implements an inference optimization method using a small, fast model to propose several tokens at once, which are then checked by a larger, more accurate model. They are surprised to find that the overall generation speed is nearly identical to using only the large model. Which of the following scenarios best explains this lack of performance improvement?