Federated Composite Optimization based on Douglas-Rachford splitting and variance reduction
Abstract
Federated Learning (FL) is a rapidly evolving distributed machine learning technique that involves a vast number of user devices. Local stochastic gradient descent (local-SGD) has received much attention and been extensively studied in FL to alleviate the communication bottleneck. However, the heterogeneity of data and the stochastic gradient induce high variance, leading to significant deviations in clients' local models. This results in slow and unstable convergence of the global model and increases communication overhead. To address this problem, recent work proposed various variance reduction schemes. In this paper, we propose a novel communication-efficient FL algorithm, Feddr-VR, which leverages the Douglas-Rachford (DR) splitting method and variance reduction to address the federated composite optimization (FCO) problem across heterogeneous data, where the objective function includes a possibly non-smooth regularization term. More specifically, for convex loss functions, we prove a sublinear convergence rate of O ( 1 k ) " role="presentation" style="position: relative;"> O ( 1 k ) O ( 1 k ) \mathcal{O}\left ( \frac{1}{k} \right ) . In addition, for strongly convex loss functions, our proposed algorithm achieves linear convergence towards a neighborhood of a first-order optimal solution. Moreover, Feddr-VR is highly practical, as it requires tuning only the step size. Finally, our numerical experiments reveal the effectiveness of our suggested algorithm in handling heterogeneous data and achieving communication efficiency.