Learn Before
Scaling Challenges in LLM Training
As a senior systems engineer, explain to a project manager why the performance gains from scaling up hardware are unlikely to be perfectly linear. Identify and describe at least three distinct types of problems or overheads that become more significant as the number of processing units increases, preventing an ideal 64x speedup in this scenario.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Communication Cost in Distributed Systems
Synchronization Costs in Distributed Systems
Fault Tolerance in Distributed Systems
Additional Scalability Factors in Distributed Training
Numerical Computation Issues in Distributed Training
A research team is training a large model on 128 processing units, and the process takes 10 days. To accelerate the training, they double the number of processing units to 256. However, the new training time is 7 days, not the expected 5 days. Which of the following statements best analyzes this outcome?
Scaling Challenges in LLM Training
Match each distributed training problem scenario with the primary underlying factor that causes it.