ChipsSeptember 16, 2026via AWS Machine Learning Blog
Fault tolerant distributed training on Amazon EKS using NVRx
Why it matters
Practitioners scaling model training on cloud infrastructure now have a battle-tested pattern to recover from hardware faults in seconds instead of hours — lowering the operational tax of large-scale training and improving cluster economics.
Key signals
- NVIDIA Resiliency Extension (NVRx) integrated with PyTorch FSDP on Amazon EKS
- 99%+ training efficiency maintained across 2–8 node H100 clusters
- Second-scale GPU fault recovery with in-process restart and ft_launcher
- Async checkpointing overlaps I/O with active training
- Benchmark data: distributed training on AWS managed Kubernetes
- NVIDIA Resiliency Extension (NVRx) integrated into PyTorch FSDP on Amazon EKS
- 99%+ training efficiency demonstrated at 2-8 node H100 clusters
- GPU fault recovery in seconds (vs. hours with traditional checkpointing)
- Async checkpointing overlaps checkpoint I/O with active training
- In-process restart and ft_launcher in-job restart mechanisms
- Benchmark: H100 multi-node configurations tested
The hook
99%+ training efficiency at scale. AWS and NVIDIA just made distributed training resilient enough to stop worrying about GPU faults.
Integrate NVIDIA Resiliency Extension (NVRx) into PyTorch FSDP training on Amazon EKS to overlap checkpoint I/O with training and recover from GPU faults in seconds. This post covers async checkpointing, in-process restart, and ft_launcher in-job restart, with H100 benchmarks at 2 to 8 nodes showing…