A team is deploying a large language model for a real-time customer support chatbot. The primary requirements are that the bot must respond quickly to user queries (low latency) and provide coherent, helpful answers (high accuracy). The team tests different settings for the parameter that controls how many potential response sequences are considered at each step of generation, with the following results:
- Setting A (Value=1): Very fast responses, but answers are often simplistic and sometimes
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Chatbot Generation Strategy Evaluation
A team is deploying a large language model for a real-time customer support chatbot. The primary requirements are that the bot must respond quickly to user queries (low latency) and provide coherent, helpful answers (high accuracy). The team tests different settings for the parameter that controls how many potential response sequences are considered at each step of generation, with the following results:
- Setting A (Value=1): Very fast responses, but answers are often simplistic and sometimes
An engineer is tuning a text generation model and plots the relationship between a key parameter, output quality, and processing time. The parameter controls the number of potential text sequences the model considers at each step. The results show that as the parameter's value increases from 1 to 4, the output quality score rises sharply. However, for values greater than 4, the quality score shows almost no further improvement. In contrast, the processing time increases steadily and significantl