Learn Before
Model Selection for a High-Traffic Application
A company is launching a new AI-powered code completion tool for software developers. They anticipate a very high volume of simultaneous users. They are testing two different language models, Model X and Model Y, which have been judged to have nearly identical accuracy and quality for the task. Performance testing yields the following data:
- Model X: Can process 400 tokens per second.
- Model Y: Can process 1,600 tokens per second.
Given that the primary business requirement is to serve a large, active user base without system slowdowns, which model is the more suitable choice? Justify your decision by explaining how the relevant performance metric informs your selection.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.4 Platform Adaptation and Performance Evaluation - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Agentic Workload Serving and Cross-Hardware Performance - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
OpenStax Psychology (2nd ed.) Textbook
Related
Model Selection for a High-Traffic Application
A company is evaluating two language model systems for a real-time translation service. System A processes 240,000 tokens in 60 seconds. System B processes 300,000 tokens in 120 seconds. Based on their processing capacity, which system is more efficient for this high-demand task and why?
A company's AI-powered customer support system experiences significant slowdowns during peak traffic times. The system can only process a limited number of user queries simultaneously, creating a bottleneck. To address this, the primary goal is to increase the number of queries the model can handle per second. Which of the following actions would most directly improve this specific performance metric?
Throughput evaluates the processing capacity of a Large Language Model as an efficiency metric.
Which two units are typically counted on a per-second basis to quantify the throughput of a Large Language Model?