Distributed Data Parallel

This poster summarizes Distributed Data Parallel as a multi-GPU training pattern: every GPU keeps a full model copy, processes different data, computes local gradients, and uses all-reduce so every process applies the same averaged update.

1. What is DDP?

Distributed Data Parallel (DDP) is a multi-GPU training strategy.

Each GPU keeps a full copy of the model, processes a different mini-batch, computes gradients locally, and synchronizes gradients across GPUs using all-reduce.

Each GPU trains on different data, but all GPUs keep the same model parameters via gradient synchronization.

2. Single-GPU Training Flow

1. Initialize
   Initialize weights (w) and bias (b)

2. Forward pass
   z = wx + b

3. Compute loss
   loss = Loss(z, y)

4. Backward pass
   Compute gradients:
   dLoss/dw
   dLoss/db

5. Optimizer step
   w = w - lr * dLoss/dw
   b = b - lr * dLoss/db

6. Repeat
   Repeat for steps and epochs

Key points:

• Backpropagation only computes gradients
• Optimizer updates parameters
• 1 epoch = one full pass through dataset

3. DDP Training Flow

1. Initialize
   Each GPU has a full model copy

2. Data split
   Each GPU gets a different mini-batch

3. Forward pass (per GPU)

4. Compute loss (per GPU)

5. Backward pass (per GPU)
   Compute local gradients

6. Gradient synchronization
   All-reduce gradients

7. Optimizer step
   Use averaged gradients

8. Repeat

DDP synchronizes gradients, not loss or weights.

4. What is All-reduce?

All-reduce aggregates values from all GPUs and returns the result to every GPU.

Example:

g_avg = (g0 + g1 + g2 + g3) / 4

Each GPU gets the same gradient -> same model update

5. All-reduce Implementations

5.1 Ring All-reduce

GPU0 -> GPU1 -> GPU2 -> GPU3 -> GPU0

• Pass gradient chunks around
• Accumulate while passing

Good for:

• Large gradient tensors
• High bandwidth efficiency
• No central bottleneck

5.2 Tree All-reduce

        GPU0
       /    \
    GPU1    GPU2

Steps:

1. Reduce (bottom -> top)
2. Broadcast (top -> bottom)

Good for:

• Fewer communication rounds
• Smaller messages / multi-node

5.3 Summary

Ring -> circle communication
Tree -> hierarchical communication

No single best choice -> depends on hardware & data size

6. NCCL in DDP

NCCL = NVIDIA Collective Communications Library

Used by PyTorch DDP for GPU communication.

DDP -> uses NCCL
NCCL -> executes all-reduce

You do NOT implement ring/tree manually.

7. Why DDP > DataParallel

DataParallel:

main GPU handles everything -> bottleneck

DDP:

one process per GPU
local compute + all-reduce

Advantages:

- no single-process bottleneck
- lower Python overhead
- faster GPU communication
- overlap communication with backward

8. How Forge Uses DDP

Forge does NOT manually write:

torch.nn.parallel.DistributedDataParallel(...)

Instead:

torchrun / SLURM
    ↓
multiple processes
    ↓
Hugging Face Trainer
    ↓
automatic DDP
    ↓
NCCL all-reduce

Single-node multi-GPU

torchrun --nproc_per_node=4 forge.py

-> 4 GPUs = 4 processes

Multi-node multi-GPU

multiple machines
each runs torchrun
all processes connect

LOCAL_RANK

Each process uses its own GPU:

LOCAL_RANK=0 -> cuda:0
LOCAL_RANK=1 -> cuda:1
...

9. DDP vs FSDP (Forge)

Strategy:

1 GPU -> off
fit in memory -> DDP
too large -> FSDP

Difference:

DDP -> full model on each GPU
FSDP -> model sharded across GPUs