Trends in LIMS

DeepSpeed

Original author(s)	Microsoft Research
Developer(s)	Microsoft
Initial release	May 18, 2020; 4 years ago (2020-05-18)
Stable release	v0.14.4 / June 21, 2024; 4 months ago (2024-06-21)
Repository	github.com/microsoft/DeepSpeed
Written in	Python, CUDA, C++
Type	Software library
License	Apache License 2.0
Website	deepspeed.ai

DeepSpeed is an open source deep learning optimization library for PyTorch.^[1]

The library is designed to reduce computing power and memory use and to train large distributed models with better parallelism on existing computer hardware.^[2][3] DeepSpeed is optimized for low latency, high throughput training. It includes the Zero Redundancy Optimizer (ZeRO) for training models with 1 trillion or more parameters.^[4] Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed under MIT License and available on GitHub.^[5]

The team claimed to achieve up to a 6.2x throughput improvement, 2.8x faster convergence, and 4.6x less communication.^[6]

• Comparison of deep learning software
• Deep learning
• Machine learning
• TensorFlow

• Rajbhandari, Samyam; Rasley, Jeff; Ruwase, Olatunji; He, Yuxiong (2019). "ZeRO: Memory Optimization Towards Training A Trillion Parameter Models". arXiv:1910.02054 [cs.LG].

• AI at Scale - Microsoft Research
• GitHub - microsoft/DeepSpeed
• ZeRO & DeepSpeed: New system optimizations enable training models with over 100 billion parameters - Microsoft Research