A company has deployed a large language model for a customer support chatbot. They observe that a small number of common questions (e.g., 'What are your business hours?') account for a large portion of the daily traffic. The company is facing challenges with both high operational costs from running the model for every query and user complaints about slow response times. Which of the following deployment-focused strategies would be most effective at directly addressing both the cost and latency issues for these frequent, repetitive queries?
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Request-Response Caching for LLM Inference
Batching in LLM Inference
Components of an LLM Inference System
Complexity of LLM Serving Systems
Choosing an LLM Optimization Strategy for Deployment
A company has deployed a large language model for a customer support chatbot. They observe that a small number of common questions (e.g., 'What are your business hours?') account for a large portion of the daily traffic. The company is facing challenges with both high operational costs from running the model for every query and user complaints about slow response times. Which of the following deployment-focused strategies would be most effective at directly addressing both the cost and latency issues for these frequent, repetitive queries?
A development team has successfully reduced their language model's size by 50% using a post-training compression method. This single change guarantees that their deployed application will now handle at least twice the user traffic with the same hardware.