In data-parallel training, after gradients are computed across multiple devices, they must be synchronized to update the model parameters. This synchronization can be implemented using centralized strategies, where all gradients are sent to a single GPU or the CPU for aggregation, or through distributed strategies, where gradients are partitioned and aggregated simultaneously across multiple GPUs to leverage the full bandwidth of hardware switches.

Parameter Synchronization Strategies

In deep neural networks, gradients are computed sequentially from the output layers back to the input layers during backpropagation. To improve distributed training performance, systems can begin synchronizing the gradients of the already-processed upper layers while the lower layers are still computing their gradients. This overlapping of communication and computation minimizes hardware idle time and accelerates the overall training iteration.

Overlapping Gradient Computation and Synchronization

Modern deep learning hardware often features bespoke network connectivity to handle large data transfers efficiently. For example, in an $$8$$-GPU server, each GPU typically connects to a host CPU via a PCIe link operating at around $$16$$ GB/s. Simultaneously, each GPU may have multiple NVLink connections to other GPUs, each capable of bidirectionally transferring data at much higher speeds (e.g., $$300$$ Gbit/s or roughly $$18$$ GB/s per direction). Because the aggregate NVLink bandwidth significantly exceeds the PCIe bandwidth, maximizing training efficiency requires specialized synchronization protocols that exploit this hardware architecture.

NVLink Connectivity in Multi-GPU Servers

Efficient multi-GPU training relies on two foundational data synchronization operations. First, parameters must be distributed to multiple devices and gradients must be attached, because without parameters it is impossible to evaluate the network on a GPU. Second, an allreduce function is required to sum parameters across multiple devices and broadcast the result back, ensuring consistency.

Claude

Data parallelism is a widely used and highly convenient strategy for distributing deep learning training across multiple GPUs. In this approach, every GPU maintains a complete replica of the model and performs the identical sequence of operations, but each processes a different subset of the training minibatch. After each minibatch, the independently computed gradients are aggregated across all GPUs to synchronize and update the model parameters. To maximize efficiency, it is highly desirable to overlap computation and communication by exchanging gradients for some parameters while others are still being computed. While data parallelism enables larger effective minibatch sizes and increases overall training throughput, it is ultimately constrained by the memory of a single GPU and does not facilitate the training of larger models.

Data Parallelism

Dive into Deep Learning

The standard delta rule for gradient descent updates a model's parameters by taking a small step in the direction of the negative loss gradient. The new parameters $$\theta_{t+1}$$ are obtained according to the formula: $$\theta_{t+1} = \theta_t - lr \cdot \frac{\partial L_{\theta_t}(\mathcal{D}_{\mathrm{mini}})}{\partial \theta_t}$$. In this equation, $$\theta_t$$ represents the latest parameters, $$lr$$ is the small step (learning rate), and the fractional term is the gradient of the loss function $$L$$ with respect to $$\theta_t$$, computed on a minibatch of training sample $$\mathcal{D}_{\mathrm{mini}}$$.

Gradient Descent Update Rule

In data parallelism, a minibatch of training sample, $$\mathcal{D}_{\mathrm{mini}}$$, is divided into $$N$$ smaller batches, which can be denoted by $$\mathcal{D}^{1},...,\mathcal{D}^{N}$$. After the division, these smaller batches are distributed to $$N$$ separate workers, each receiving one corresponding batch, allowing them to work at the same time.

Set of Distributed Data Batches in Data Parallelism

Under optimal conditions, data parallelism can significantly accelerate the training process. When worker coordination is efficient and communication overhead is negligible, the training speed can increase by a factor of nearly $$N$$, where $$N$$ is the number of workers. This represents a near-linear speed-up.

Ideal Speed-up in Data Parallelism

A team is training a neural network using a technique where a large batch of data is split equally among 8 machines. Each machine has a full, identical copy of the network model. During a training step, each machine processes its portion of the data and calculates a set of proposed parameter updates. Given this setup, what is the most critical subsequent action to ensure the entire system learns effectively from the full batch of data?

In distributed training using data parallelism, the gradient of the loss function, $$L$$, with respect to the parameters, $$\theta_t$$, for a complete mini-batch, $$\mathcal{D}_{\mathrm{mini}}$$, is computed by summing the gradients from multiple workers. Each worker calculates the gradient on a separate partition of the mini-batch, denoted as $$\mathcal{D}^1, \mathcal{D}^2, \dots, \mathcal{D}^N$$. This aggregation of gradients is represented by the formula:

$$\frac{\partial L_{\theta_t}(\mathcal{D}_{\mathrm{mini}})}{\partial \theta_t} = \underbrace{\frac{\partial L_{\theta_t}(\mathcal{D}^1)}{\partial \theta_t}}_{\textrm{worker 1}} + \underbrace{\frac{\partial L_{\theta_t}(\mathcal{D}^2)}{\partial \theta_t}}_{\textrm{worker 2}} + \cdots + \underbrace{\frac{\partial L_{\theta_t}(\mathcal{D}^N)}{\partial \theta_t}}_{\textrm{worker } N}$$

This allows for parallel computation, significantly speeding up the training process for large models.

Distributed Gradient Calculation

A single training step is performed using a technique where a mini-batch of data is processed in parallel across multiple machines. Each machine holds a complete copy of the model. Arrange the following events in the correct chronological order for one such training step.

A machine learning team is training a large neural network on a massive dataset. To accelerate the process, they employ a strategy where the training data is split across 16 GPUs. Each GPU holds a complete copy of the model and processes its own subset of the data. After each forward and backward pass, the results from all GPUs are combined before updating the model's parameters. The team observes that while using 8 GPUs provided a nearly 8x speed-up compared to a single GPU, scaling to 16 GPUs only resulted in a 10x total speed-up. Based on the principles of the training strategy described, what is the most likely bottleneck causing this diminishing return in performance when scaling from 8 to 16 GPUs?

A research team is planning to train a very large neural network. They have access to a cluster of 8 powerful machines, but they discover that the model's full set of parameters is too large to fit into the memory of any single machine. A junior member of the team suggests using a training method where the data is split among the machines to speed up the process. Evaluate this suggestion. Is this a viable strategy for their specific problem? Justify your conclusion based on the fundamental requirements of this training method.

Evaluating a Training Strategy

Your team must train a 30B-parameter LLM on a sing...

You are on-call for an internal LLM training platf...

Your team is training a 70B-parameter LLM on 8 GPU...

You’re advising an internal platform team that mus...

You are the tech lead for training a new LLM that cannot fit on a single GPU due to parameter/activation memory, but leadership also expects near-linear throughput scaling when moving from 8 to 32 GPUs. Your cluster has 32 identical GPUs connected with high-bandwidth intra-node links and slower inter-node links. You must choose a distributed training approach that combines (as needed) data parallelism, model parallelism, pipeline parallelism (with micro-batching), and mixed precision training.

Write a recommendation memo that proposes a concrete parallelism/mixed-precision strategy and justifies it. Your memo must: (1) explain how your design resolves the single-GPU out-of-memory issue, (2) explain where and why gradient synchronization/communication happens and how it affects scaling, (3) explain how pipeline micro-batching changes device utilization compared with naive layer-splitting, and (4) explain how mixed precision improves speed/memory while still keeping training numerically stable (e.g., what stays in higher precision and why). Conclude by identifying the most likely bottleneck that will prevent perfect 32x scaling in your design and one mitigation you would try first.

Designing a Distributed Training Plan Under Memory, Throughput, and Stability Constraints

You are the on-call ML engineer for a corporate LLM fine-tuning job running on 8 GPUs (each 40 GB). The model is too large to fit on a single GPU in full precision, so the team split the model across 4 GPUs in sequential stages (a pipeline). To increase throughput, they also run 2 identical pipeline replicas (so all 8 GPUs are used) and split each global mini-batch across the 2 replicas. They enabled mixed precision so most compute uses FP16, while a master copy of weights is kept in FP32 for updates.

After several hours, the run shows two problems: (1) training loss becomes unstable and occasionally spikes to NaN; (2) GPU utilization is uneven—some GPUs are frequently idle even though the input pipeline is not the bottleneck.

Write a postmortem-style response that (a) identifies the most plausible root causes that connect the chosen parallelism strategy (data parallel across replicas + model/pipeline parallel within a replica) with mixed precision behavior, and (b) proposes a concrete redesign of the training step to address BOTH numerical stability and utilization. Your answer must explicitly explain the interactions/tradeoffs among gradient aggregation across replicas, micro-batching in the pipeline, and FP16/FP32 precision choices (e.g., where precision should be used and why), and justify why your redesign would reduce NaNs while improving end-to-end throughput.

Postmortem and Redesign of a Distributed LLM Training Run with Divergence and Low GPU Utilization

You are the on-call ML engineer for a team training a 30B-parameter LLM on a 64-GPU cluster (8 nodes × 8 GPUs). The model does not fit on a single GPU, so the team shards the model across 4 GPUs per replica (model parallelism) and uses pipeline parallelism with micro-batches to keep those 4 GPUs busy. They then replicate this 4-GPU pipeline across the remaining GPUs using data parallelism, synchronizing gradients across replicas each step. To reduce memory and increase throughput, they enable mixed precision (FP16 compute with FP32 master weights).

After a change request to “increase throughput,” the team doubles the number of data-parallel replicas (more pipelines in parallel) and also increases the number of micro-batches per step. Throughput improves, but two problems appear: (1) scaling efficiency drops sharply (adding replicas yields little additional speed), and (2) training becomes less stable (loss occasionally spikes or diverges).

Write an analysis that identifies the most likely root causes of BOTH problems and proposes a concrete mitigation plan. Your answer must explicitly connect how data parallel gradient synchronization, pipeline micro-batching, model sharding, and mixed precision interact (e.g., communication volume/frequency, pipeline bubbles/latency hiding, effective batch size and update frequency, and numerical stability during gradient aggregation). Conclude by recommending one revised configuration (at a high level) and justify the tradeoffs you are making.

Diagnosing a Scaling Regression in Hybrid Parallel LLM Training

You are the on-call ML platform lead for a company training a 30B-parameter transformer. You have access to two clusters:

- Cluster A: 8 GPUs/node, 80 GB VRAM each, fast NVLink within node, 200 Gbps inter-node network.
- Cluster B: 8 GPUs/node, 40 GB VRAM each, slower interconnect within node, 100 Gbps inter-node network.

The team’s current setup uses pure data parallelism with FP16 mixed precision (FP16 compute, FP32 master weights). On Cluster A it trains stably but is slower than expected; on Cluster B it frequently hits out-of-memory errors unless the global batch size is reduced so much that throughput collapses.

You are asked to propose ONE distributed training configuration that can run on both clusters with minimal code divergence. Your proposal must specify how you will combine (a) data parallelism, (b) model parallelism, (c) pipeline parallelism (including whether you will use micro-batches), and (d) mixed precision choices, and it must justify the key tradeoffs you are making between memory fit, communication overhead, device utilization, and numerical stability.

What configuration do you recommend, and why is it the best compromise given the two clusters’ constraints?

Stabilizing and Scaling an LLM Training Job Across Two GPU Clusters

You are the tech lead for an internal LLM training platform. Your team is moving a 30B-parameter transformer pretraining job to a new cluster: 32 GPUs total, each with 24 GB VRAM, connected with high-bandwidth interconnect. The model does not fit on a single GPU in FP32, and the team wants to maximize tokens/sec while keeping training stable.

Two candidate configurations are proposed:

A) Pure data parallelism across all 32 GPUs, using mixed precision (FP16 compute with FP32 master weights) and gradient all-reduce every step.

B) A hybrid approach: split the model into 4 sequential pipeline stages (pipeline parallelism) with micro-batches, use model parallelism within each stage across 2 GPUs to fit the largest layers, and then use data parallelism across the remaining replicas; also use mixed precision (FP16 compute with FP32 master weights).

During a pilot run, the team observes:
- With A, the job cannot start due to out-of-memory errors even at small batch sizes.
- With B, the job starts, but tokens/sec is lower than expected and some GPUs show periodic idle gaps.

As the decision-maker, which configuration should you choose and what specific adjustment(s) would you make to address the observed issues while preserving numerical stability? In your answer, explicitly connect (1) why the memory constraint rules out or enables a strategy, (2) how data/model/pipeline parallelism interact to affect utilization and communication, and (3) how mixed precision changes both memory headroom and stability requirements.

Choosing a Distributed Training Configuration After a Hardware Refresh

You are the on-call ML engineer for a corporate LLM fine-tuning job that must finish within a weekend. The model is 30B parameters and does not fit in the memory of a single 80GB GPU in full precision. You have access to 8 identical 80GB GPUs connected with high-bandwidth interconnect. The team’s first attempt used pure data parallelism (one full model replica per GPU) and failed with out-of-memory errors. A second attempt split the model across 4 GPUs (layer-wise model parallelism) and ran, but GPU utilization was low because only one stage was busy at a time; throughput was far below target. A third attempt enabled mixed precision (FP16 compute) and ran faster, but training became numerically unstable unless the learning rate was reduced so much that the weekend deadline was missed.

As the incident owner, propose a single revised distributed training design that uses (a) an appropriate combination of data parallelism, model parallelism, and pipeline parallelism, and (b) mixed precision in a way that improves both memory feasibility and throughput while maintaining numerical stability. Your answer must explicitly justify: (1) how your design resolves the original OOM issue, (2) how it avoids the low-utilization problem seen in the sequential model split, and (3) what specific mixed-precision practice(s) you would use to reduce instability while keeping most of the speed/memory benefits.

Selecting a Hybrid Parallelism + Mixed-Precision Strategy for a Memory-Bound LLM Training Run

When distributing training across $$k$$ GPUs using data parallelism, it is standard practice to scale the overall minibatch size by a factor of $$k$$. This ensures that each individual GPU processes the same amount of data, and performs an equivalent computational workload, as it would if training on a single GPU. Because this scaling considerably increases the effective minibatch size—such as a $$16$$-fold increase on a $$16$$-GPU server—the learning rate must typically be increased proportionally to maintain stable and efficient optimization.

Minibatch Scaling in Data Parallelism

Applying batch normalization during multi-GPU data parallelism requires specific architectural adjustments. Because the global minibatch is distributed across multiple devices, computing the exact normalization statistics across the entire batch would necessitate costly cross-device synchronization. A practical solution is to maintain a separate batch normalization coefficient for each GPU, allowing each device to calculate its own mean and variance statistics locally based solely on its assigned subset of the minibatch data.

Batch Normalization in Data Parallelism

In a data parallelism setup, the training process follows a specific sequence during each iteration. First, a random minibatch of training data is split into $$k$$ equal portions and distributed across the $$k$$ available GPUs. Each GPU then independently calculates the loss and the local gradients of the model parameters using its assigned data subset. These local gradients from all $$k$$ GPUs are subsequently aggregated (via an allreduce operation) to compute the overall minibatch stochastic gradient. This aggregated gradient is redistributed to every GPU, and each GPU uses it to independently update its complete set of model parameters, ensuring synchronization across the system.

Data Parallelism Training Process

Data Synchronization in Multi-GPU Training

The effectiveness of multi-GPU data parallelism depends critically on the ratio of computation time to synchronization overhead. When a model is computationally lightweight (e.g., LeNet), the time spent on the forward pass and gradient computation is comparable to or smaller than the time required for cross-device parameter synchronization and Python scheduling overhead. In such cases, adding more GPUs yields no meaningful speedup. Conversely, when a model is sufficiently complex (e.g., ResNet-18), the per-device computation time dominates the synchronization cost, making the parallelization overhead relatively negligible and enabling significant scalability improvements as more GPUs are added.

Learn Before

Related

Learn After