More Like this

1. FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tiers

   https://doi.org/10.1145/3458817.3476211  

   Chai, Zheng ; Chen, Yujing ; Anwar, Ali ; Zhao, Liang ; Cheng, Yue ; Rangwala, Huzefa ( November 2021 , The International Conference for High Performance Computing, Networking, Storage, and Analysis)

   Federated learning (FL) involves training a model over massive distributed devices, while keeping the training data localized and private. This form of collaborative learning exposes new tradeoffs among model convergence speed, model accuracy, balance across clients, and communication cost, with new challenges including: (1) straggler problem—where clients lag due to data or (computing and network) resource heterogeneity, and (2) communication bottleneck—where a large number of clients communicate their local updates to a central server and bottleneck the server. Many existing FL methods focus on optimizing along only one single dimension of the tradeoff space. Existing solutions use asynchronous model updatingmore »or tiering-based, synchronous mechanisms to tackle the straggler problem. However, asynchronous methods can easily create a communication bottleneck, while tiering may introduce biases that favor faster tiers with shorter response latencies. To address these issues, we present FedAT, a novel Federated learning system with Asynchronous Tiers under Non-i.i.d. training data. FedAT synergistically combines synchronous, intra-tier training and asynchronous, cross-tier training. By bridging the synchronous and asynchronous training through tiering, FedAT minimizes the straggler effect with improved convergence speed and test accuracy. FedAT uses a straggler-aware, weighted aggregation heuristic to steer and balance the training across clients for further accuracy improvement. FedAT compresses uplink and downlink communications using an efficient, polyline-encoding-based compression algorithm, which minimizes the communication cost. Results show that FedAT improves the prediction performance by up to 21.09% and reduces the communication cost by up to 8.5×, compared to state-of-the-art FL methods.« less
2. Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacy

   Xinwei Zhang ; Xiangyi Chen ; Mingyi Hong ; Zhiwei Steven Wu ; Jinfeng Yi ( July 2022 , International Conference on Machine Learning)

   Providing privacy protection has been one of the primary motivations of Federated Learning (FL). Recently, there has been a line of work on incorporating the formal privacy notion of differential privacy with FL. To guarantee the client-level differential privacy in FL algorithms, the clients’ transmitted model updates have to be clipped before adding privacy noise. Such clipping operation is substantially different from its counterpart of gradient clipping in the centralized differentially private SGD and has not been well-understood. In this paper, we first empirically demonstrate that the clipped FedAvg can perform surprisingly well even with substantial data heterogeneity when trainingmore »neural networks, which is partly because the clients’ updates become similar for several popular deep architectures. Based on this key observation, we provide the convergence analysis of a differential private (DP) FedAvg algorithm and highlight the relationship between clipping bias and the distribution of the clients’ updates. To the best of our knowledge, this is the first work that rigorously investigates theoretical and empirical issues regarding the clipping operation in FL algorithms.« less
3. Exploiting Federated Learning Technique to Recognize Human Activities in Resource-Constrained Environment

   Ahmed Imteaj, Raghad Alabagi ( December 2021 , The 13th International Conference on Intelligent Human Computer Interaction (IHCI-2021))

   The conventional machine learning (ML) and deep learning (DL) methods use large amount of data to construct desirable prediction models in a central fusion center for recognizing human activities. However, such model training encounters high communication costs and leads to privacy infringement. To address the issues of high communication overhead and privacy leakage, we employed a widely popular distributed ML technique called Federated Learning (FL) that generates a global model for predicting human activities by combining participated agents’ local knowledge. The state-of-the-art FL model fails to maintain acceptable accuracy when there is a large number of unreliable agents who canmore »infuse false model, or, resource-constrained agents that fails to perform an assigned computational task within a given time window. We developed an FL model for predicting human activities by monitoring agent’s contributions towards model convergence and avoiding the unreliable and resource-constrained agents from training. We assign a score to each client when it joins in a network and the score is updated based on the agent’s activities during training. We consider three mobile robots as FL clients that are heterogeneous in terms of their resources such as processing capability, memory, bandwidth, battery-life and data volume. We consider heterogeneous mobile robots for understanding the effects of real-world FL setting in presence of resource-constrained agents. We consider an agent unreliable if it repeatedly gives slow response or infuses incorrect models during training. By disregarding the unreliable and weak agents, we carry-out the local training of the FL process on selected agents. If somehow, a weak agent is selected and started showing straggler issues, we leverage asynchronous FL mechanism that aggregate the local models whenever it receives a model update from the agents. Asynchronous FL eliminates the issue of waiting for a long time to receive model updates from the weak agents. To the end, we simulate how we can track the behavior of the agents through a reward-punishment scheme and present the influence of unreliable and resource-constrained agents in the FL process. We found that FL performs slightly worse than centralized models, if there is no unreliable and resource-constrained agent. However, as the number of malicious and straggler clients increases, our proposed model performs more effectively by identifying and avoiding those agents while recognizing human activities as compared to the stateof-the-art FL and ML approaches.« less
4. Emergent complex quantum networks in continuous-variables non-Gaussian states

   Mattia Walschaers, Nicolas Treps ( July 2021 , ArXivorg)

   Large multipartite quantum systems tend to rapidly reach extraordinary levels of complexity as their number of constituents and entanglement links grow. Here we use complex network theory to study a class of continuous variables quantum states that present both multipartite entanglement and non-Gaussian statistics. In particular, the states are built from an initial imprinted cluster state created via Gaussian entangling operations according to a complex network structure. To go beyond states that can be easily simulated via classical computers we engender non-Gaussian statistics via multiple photon subtraction operations. We then use typical networks measures, the degree and clustering, to characterizemore »the emergent complex network of photon-number correlations after photon subtractions. We show that, in contrast to regular clusters, in the case of imprinted complex network structures the emergent correlations are strongly affected by photon subtraction. On the one hand, we unveil that photon subtraction universally increases the average photon-number correlations, regardless of the imprinted network structure. On the other hand, we show that the shape of the distributions in the emergent networks after subtraction is greatly influenced by the structure of the imprinted network, as witnessed by their higher-moments. Thus for the field of network theory, we introduce a new class of networks to study. At the same time for the field of continuous variable quantum states, this work presents a new set of practical tools to benchmark systems of increasing complexity.« less
5. Federated Multi-Armed Bandits

   Shi, C ; Shen, C ( May 2021 , Proceedings of the AAAI Conference on Artificial Intelligence)

   Federated multi-armed bandits (FMAB) is a new bandit paradigm that parallels the federated learning (FL) framework in supervised learning. It is inspired by practical applications in cognitive radio and recommender systems, and enjoys features that are analogous to FL. This paper proposes a general framework of FMAB and then studies two specific federated bandit models. We first study the approximate model where the heterogeneous local models are random realizations of the global model from an unknown distribution. This model introduces a new uncertainty of client sampling, as the global model may not be reliably learned even if the finite localmore »models are perfectly known. Furthermore, this uncertainty cannot be quantified a priori without knowledge of the suboptimality gap. We solve the approximate model by proposing Federated Double UCB (Fed2-UCB), which constructs a novel “double UCB” principle accounting for uncertainties from both arm and client sampling. We show that gradually admitting new clients is critical in achieving an O(log(T)) regret while explicitly considering the communication loss. The exact model, where the global bandit model is the exact average of heterogeneous local models, is then studied as a special case. We show that, somewhat surprisingly, the order-optimal regret can be achieved independent of the number of clients with a careful choice of the update periodicity. Experiments using both synthetic and real-world datasets corroborate the theoretical analysis and demonstrate the effectiveness and efficiency of the proposed algorithms.« less