More Like this

1. Overlap Local-SGD: An Algorithmic Approach to Hide Communication Delays in Distributed SGD

   https://doi.org/10.1109/ICASSP40776.2020.9053834  

   Wang, Jianyu ; Liang, Hao ; Joshi, Gauri ( May 2020 , IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP))

   null (Ed.)

   Distributed stochastic gradient descent (SGD) is essential for scaling the machine learning algorithms to a large number of computing nodes. However, the infrastructures variability such as high communication delay or random node slowdown greatly impedes the performance of distributed SGD algorithm, especially in a wireless system or sensor networks. In this paper, we propose an algorithmic approach named Overlap Local-SGD (and its momentum variant) to overlap communication and computation so as to speedup the distributed training procedure. The approach can help to mitigate the straggler effects as well. We achieve this by adding an anchor model on each node. After multiple local updates, locally trained models will be pulled back towards the synchronized anchor model rather than communicating with others. Experimental results of training a deep neural network on CIFAR-10 dataset demonstrate the effectiveness of Overlap Local-SGD. We also provide a convergence guarantee for the proposed algorithm under non-convex objective functions. 

   more » « less
2. Open Problem: Can Single-Shuffle SGD be Better than Reshuffling SGD and GD?

   Yun, Chulhee ; Sra, Suvrit ; Jadbabaie, Ali ( January 2021 , Proceedings of Machine Learning Research)
3. SQuARM-SGD: Communication-Efficient Momentum SGD for Decentralized Optimization

   https://doi.org/10.1109/JSAIT.2021.3103920  

   Singh, Navjot ; Data, Deepesh ; George, Jemin ; Diggavi, Suhas ( September 2021 , IEEE Journal on Selected Areas in Information Theory)
4. Cooperative SGD: A Unified Framework for the Design and Analysis of Local-Update SGD Algorithms

   Wang, Jianyu ; Joshi, Gauri ( January 2021 , Journal of machine learning research)

   When training machine learning models using stochastic gradient descent (SGD) with a large number of nodes or massive edge devices, the communication cost of synchronizing gradients at every iteration is a key bottleneck that limits the scalability of the system and hinders the benefit of parallel computation. Local-update SGD algorithms, where worker nodes perform local iterations of SGD and periodically synchronize their local models, can effectively reduce the communication frequency and save the communication delay. In this paper, we propose a powerful framework, named Cooperative SGD, that subsumes a variety of local-update SGD algorithms (such as local SGD, elastic averaging SGD, and decentralized parallel SGD) and provides a unified convergence analysis. Notably, special cases of the unified convergence analysis provided by the cooperative SGD framework yield 1) the first convergence analysis of elastic averaging SGD for general non-convex objectives, and 2) improvements upon previous analyses of local SGD and decentralized parallel SGD. Moreover, we design new algorithms such as elastic averaging SGD with overlapped computation and communication, and decentralized periodic averaging which are shown to be 4x or more faster than the baseline in reaching the same training loss. 

   more » « less
5. Cooperative SGD: A Unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms

   Wang, Jianyu ; Joshi, Gauri ( June 2019 , ICML Workshop on Coding Theory for Machine Learning)

   Communication-efficient SGD algorithms, which allow nodes to perform local updates and periodically synchronize local models, are highly effective in improving the speed and scalability of distributed SGD. However, a rigorous convergence analysis and comparative study of different communication-reduction strategies remains a largely open problem. This paper presents a unified framework called Cooperative SGD that subsumes existing communication-efficient SGD algorithms such as periodic-averaging, elastic-averaging, and decentralized SGD. By analyzing Cooperative SGD, we provide novel convergence guarantees for existing algorithms. Moreover, this framework enables us to design new communication-efficient SGD algorithms that strike the best balance between reducing communication overhead and achieving fast error convergence with a low error floor. 

   more » « less