The size of machine learning models (i.e., number of parameters) and of the datasets upon which they are trained are rapidly growing. To support the development of such large models, ML practitioners must often leverage distributed training algorithms. Researchers have developed new techniques to remove the network communication bottleneck through communication compression, communication decentralization, communication delay, and asynchronization. To make these techniques easier to implement, researchers from ETH Zurich and Kuaishou Technology have OSS’ed Bagua. Bagua, which is written in Rust, is designed to support SOTA relaxation techniques for distributed training.