GPU wiring can halve training speed—hardware matters as much as parallelism
The article measures DDP, FSDP and DeepSpeed ZeRO on four A100 80GB GPUs and shows that mismatched NVLink wiring on H200 nodes can slow training by up to 2×. It proves that choosing a parallelism strategy without accounting for inter‑GPU bandwidth can waste hardware and time.