Fault tolerant distributed training on Amazon EKS using NVRx
99%+ training efficiency at scale. AWS and NVIDIA just made distributed training resilient enough to stop worrying about GPU faults.
Why it matters
Practitioners scaling model training on cloud infrastructure now have a battle-tested pattern to recover from hardware faults in seconds instead of hours — lowering the operational tax of large-scale training and improving cluster economics.
The key facts
11 to knowNVIDIA Resiliency Extension (NVRx) integrated with PyTorch FSDP on Amazon EKS
99%+ training efficiency maintained across 2–8 node H100 clusters
Second-scale GPU fault recovery with in-process restart and ft_launcher
Async checkpointing overlaps I/O with active training
Benchmark data: distributed training on AWS managed Kubernetes
NVIDIA Resiliency Extension (NVRx) integrated into PyTorch FSDP on Amazon EKS
99%+ training efficiency demonstrated at 2-8 node H100 clusters
GPU fault recovery in seconds (vs. hours with traditional checkpointing)
Async checkpointing overlaps checkpoint I/O with active training
In-process restart and ft_launcher in-job restart mechanisms
Benchmark: H100 multi-node configurations tested
Go to the source
AWS Machine Learning Blogaws.amazon.com
Publisher excerpt: Integrate NVIDIA Resiliency Extension (NVRx) into PyTorch FSDP training on Amazon EKS to overlap checkpoint I/O with training and recover from GPU faults in seconds. This post covers async checkpointing, in-process restart, and ft_launcher in-job restart, with H100 benchmarks at 2 to 8 nodes…