A PyTorch extension that facilitates decentralized data parallel training. [ICLR'25] From Promise to Practice: Realizing High-performance Decentralized Training. - View it on GitHub