Private Replicas, Identical Updates

Start with the right picture

The common wrong picture is one model object spread across several GPUs. The correct picture is several isolated processes.

The distinction between the pieces matters:

At the start of a two-rank job, the state is private:

Rank 0 / Process 0                 Rank 1 / Process 1
weight = 1.0                       weight = 1.0
local gradient = 2.0               local gradient = 6.0

If both processes stepped on their own local gradients, the replicas would immediately diverge. The real question is not "how can two GPUs share one model object?" It is:

A two-process update

A small CPU example that runs on a laptop. GPU jobs use NCCL, but the core behavior is the same.

This example captures the whole pattern: private replicas, a collective reduction, and identical local updates — no GPU required. It uses Gloo so it runs anywhere.

import torch
import torch.distributed as dist

dist.init_process_group(backend="gloo")

rank = dist.get_rank()
world_size = dist.get_world_size()

# Each process owns this tensor in its own address space.
weight = torch.tensor([1.0])

# Each process has a different local gradient.
local_grad = torch.tensor([2.0 if rank == 0 else 6.0])

print(
    f"rank={rank}: weight={weight.item()}, local_grad={local_grad.item()}",
    flush=True,
)

# Sum every rank's contribution into every rank's local tensor.
dist.all_reduce(local_grad, op=dist.ReduceOp.SUM)

# DDP's usual semantic result is the mean gradient.
mean_grad = local_grad / world_size

# Each process updates its own local weight.
weight -= 0.1 * mean_grad

print(
    f"rank={rank}: mean_grad={mean_grad.item()}, updated_weight={weight.item()}",
    flush=True,
)

dist.destroy_process_group()

Launch two worker processes:

torchrun --standalone --nproc-per-node=2 toy_ddp.py

The output order is nondeterministic, but the logical result is:

rank=0: weight=1.0, local_grad=2.0
rank=1: weight=1.0, local_grad=6.0

rank=0: mean_grad=4.0, updated_weight=0.6
rank=1: mean_grad=4.0, updated_weight=0.6

Nothing was shared in memory. Both processes independently changed their private weight tensor from 1.0 to 0.6.

torchrun gives workers coordinates

A launcher, not a collective. It starts processes and tells them how to find each other.

torchrun is a launcher. It starts worker processes and gives them the information needed to discover one another.

torchrun does not synchronize gradients. It does not create a DDP wrapper. It does not make separate Python processes share memory. It launches workers and gives them a common rendezvous configuration.

init_process_group turns coordinates into membership

Until this call, each process is isolated Python. After it, each belongs to a communication group.

Before initialization, each worker is just an isolated Python process.

dist.init_process_group(backend="gloo")

After that call, each process has joined the same logical process group. Calls that do not specify group=... use this default group.

A process group is a membership contract: it defines which ranks are expected to participate in collectives together.

The important distinction:

torchrun                  → starts workers and gives them coordinates
init_process_group()      → creates group membership
collective communication  → synchronizes private state among group members

The rendezvous is setup. It is not the data path for every gradient update.

What all_reduce actually does

Reduce every rank's contribution. Write the result back to every participating rank. Still private memory.

The name is useful:

• reduce: combine every rank's contribution.
• all: write the reduced result back to every participating rank.

For the toy example:

Before all_reduce:
  rank 0 owns local_grad = 2
  rank 1 owns local_grad = 6

After all_reduce(SUM):
  rank 0 owns local_grad = 8
  rank 1 owns local_grad = 8

After dividing by world_size:
  rank 0 owns mean_grad = 4
  rank 1 owns mean_grad = 4

After the local optimizer step:
  rank 0 owns weight = 0.6
  rank 1 owns weight = 0.6

Each result still lives in private local memory. all_reduce does not create a shared tensor.

The example uses SUM followed by division so the math is visible. DDP normally gives each rank the mean gradient during backward.

A collective is a group-wide protocol step

Not an RPC. Not shared memory. Every member must eventually arrive at a compatible call.

A collective is not an RPC where rank 0 asks another rank for data. Every participating rank must enter compatible collective calls in the same order. They do not need to arrive at the same wall-clock instant, but the protocol must match:

• same process group,
• compatible operation,
• compatible tensor metadata,
• matching collective order.

This is a correctness requirement, not a performance optimization.

if rank == 0:
    dist.all_reduce(local_grad)  # rank 1 never arrives

Depending on backend and timeout settings, the job may hang, time out, or report a communication error.

The rule is simple:

A mismatched collective sequence is enough to break the job even when each local model's math is fine. The contract rules participation order — it does not care whether your loss function is correct.

DDP automates this schedule during backward

The manual toy and real DDP share one plan: synchronize gradients, then update local replicas.

Real DDP places that logic into the normal training loop:

local forward
→ local loss
→ backward computes local gradients
→ DDP hooks reduce corresponding gradients
→ every rank receives the same averaged gradients
→ every local optimizer calls step()

At construction, DDP broadcasts model state from rank 0 so replicas begin aligned. With the default buffer behavior, it also synchronizes model buffers before forward passes.

During backward, DDP uses autograd hooks and gradient buckets to overlap communication with computation where possible. After the reductions complete, every rank has matching .grad values for corresponding parameters.

The optimizer is still local. Every rank owns an optimizer instance and calls optimizer.step() on its own replica.

That is why the replicas remain aligned:

More machines change placement, not meaning

Launch coordinates change. The meaning of a collective does not.

A two-node job still follows the same model. Each node launches its local workers; all workers join one process group.

torchrun \
  --nnodes=2 \
  --node-rank=0 \
  --nproc-per-node=2 \
  --master-addr=10.0.0.1 \
  --master-port=29500 \
  toy_ddp.py

torchrun \
  --nnodes=2 \
  --node-rank=1 \
  --nproc-per-node=2 \
  --master-addr=10.0.0.1 \
  --master-port=29500 \
  toy_ddp.py

MASTER_ADDR and MASTER_PORT support setup and rendezvous. They do not turn the master node into a permanent parameter server on the gradient data path.

Across machines, the invariant remains:

private worker state
+ one process group
+ ordered compatible collectives
= synchronized updates

Where this model stops

DDP replicates model parameters, gradients, and usually optimizer state across ranks. It is often the right first distributed-training model, but replication eventually becomes a memory limit.

FSDP changes the memory layout by sharding model state. Tensor parallelism partitions tensor operations. Those systems are different, but they still rely on explicit groups and ordered communication contracts.