Based on the provided scenario, identify the most likely reason for the system's slow performance and propose a specific adjustment to the generation process to address it. What is the primary trade-off associated with your proposed adjustment?

Google

A primary drawback of search scaling is the inherent increase in computational overhead. Expanding the search space, for example by increasing the beam width in beam search, directly results in higher memory usage and extended inference times.

Computational Costs of Search Scaling

In practice, applying search scaling often encounters a point of diminishing returns. Beyond this point, further expanding the search space yields only marginal improvements in the quality of the output, while the computational expense increases significantly.

Diminishing Returns in Search Scaling

Optimizing Inference Performance

An engineer modifies a language model's inference procedure to evaluate a significantly larger number of potential output sequences at each generation step, aiming to enhance the final output quality. What is the most direct and unavoidable trade-off associated with this modification?

A developer is tuning a text generation system. They have two configurations:

*   **Configuration A:** At each step of generating text, the system considers only the 3 most likely next words.
*   **Configuration B:** At each step, the system considers the 100 most likely next words.

Describe two specific computational resources that will be more heavily consumed by Configuration B compared to Configuration A, and briefly explain why for each.

Learn Before

Related