Learn Before
Consider a computational process distributed across four sequential stages (S1, S2, S3, S4), each on a different device. A large data batch is partitioned into smaller, uniform 'micro-batches' (MB1, MB2, MB3, etc.) to be processed in a continuous flow. At a particular point in time, device S3 has just completed its work on MB1 and passed it to S4. What is the activity of device S1 at this exact moment, assuming the pipeline is running efficiently and has been for some time?
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Trade-off of Micro-batch Size in Pipeline Parallelism
Consider a computational process distributed across four sequential stages (S1, S2, S3, S4), each on a different device. A large data batch is partitioned into smaller, uniform 'micro-batches' (MB1, MB2, MB3, etc.) to be processed in a continuous flow. At a particular point in time, device S3 has just completed its work on MB1 and passed it to S4. What is the activity of device S1 at this exact moment, assuming the pipeline is running efficiently and has been for some time?
Pipeline Efficiency Analysis
Mechanism of Utilization Improvement in Pipelined Systems