Learn Before
Optimizing an LLM Inference System
Based on the typical two-component structure of a practical inference system, identify the component that is likely missing or poorly implemented in the startup's architecture. Explain how introducing this component and its primary function would address the observed issues of low hardware utilization and low overall throughput.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Scheduler in LLM Inference Systems
Inference Engine in LLM Systems
Request Processing Workflow in LLM Inference
A team is optimizing their system for serving a large language model. They observe that during peak traffic, many user requests fail with a timeout error before the model begins processing them. At the same time, monitoring shows that the hardware responsible for the model's computations is frequently idle. Based on this scenario, which of the following actions would most directly target the likely cause of this bottleneck?
A system designed to serve a large language model is composed of distinct parts, each with a specific job. Match each component with its primary responsibility within the system.
Optimizing an LLM Inference System
LLM Inference Architecture with Scheduling