More Like this

1. Distributed Learning without Distress: Privacy-Preserving Empirical Risk Minimization

   Bargav Jayaraman, Lingxiao Wang (December 2018, Advances in neural information processing systems)

   Distributed learning allows a group of independent data owners to collaboratively learn a model over their data sets without exposing their private data. We present a distributed learning approach that combines differential privacy with secure multi-party computation. We explore two popular methods of differential privacy, output perturbation and gradient perturbation, and advance the state-of-the-art for both methods in the distributed learning setting. In our output perturbation method, the parties combine local models within a secure computation and then add the required differential privacy noise before revealing the model. In our gradient perturbation method, the data owners collaboratively train a global model via an iterative learning algorithm. At each iteration, the parties aggregate their local gradients within a secure computation, adding sufficient noise to ensure privacy before the gradient updates are revealed. For both methods, we show that the noise can be reduced in the multi-party setting by adding the noise inside the secure computation after aggregation, asymptotically improving upon the best previous results. Experiments on real world data sets demonstrate that our methods provide substantial utility gains for typical privacy requirements. 

   more » « less
2. Learning Graphical Models from a Distributed Stream

   https://doi.org/10.1109/ICDE.2018.00071  

   Zhang, Yu; Tirthapura, Srikanta; Cormode, Graham (April 2018, Proceedings of the IEEE 34th International Conference on Data Engineering (ICDE))

   A current challenge for data management systems is to support the construction and maintenance of machine learning models over data that is large, multi-dimensional, and evolving. While systems that could support these tasks are emerging, the need to scale to distributed, streaming data requires new models and algorithms. In this setting, as well as computational scalability and model accuracy, we also need to minimize the amount of communication between distributed processors, which is the chief component of latency. We study Bayesian Networks, the workhorse of graphical models, and present a communication-efficient method for continuously learning and maintaining a Bayesian network model over data that is arriving as a distributed stream partitioned across multiple processors. We show a strategy for maintaining model parameters that leads to an exponential reduction in communication when compared with baseline approaches to maintain the exact MLE (maximum likelihood estimation). Meanwhile, our strategy provides similar prediction errors for the target distribution and for classification tasks. 

   more » « less
3. Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing

   Khodak, M.; Tu, R.; Li, T.; Li, L.; Balcan, M.-F.; Smith, V.; Talwalkar, A. (January 2021, Advances in neural information processing systems)

   Tuning hyperparameters is a crucial but arduous part of the machine learning pipeline. Hyperparameter optimization is even more challenging in federated learning, where models are learned over a distributed network of heterogeneous devices; here, the need to keep data on device and perform local training makes it difficult to efficiently train and evaluate configurations. In this work, we investigate the problem of federated hyperparameter tuning. We first identify key challenges and show how standard approaches may be adapted to form baselines for the federated setting. Then, by making a novel connection to the neural architecture search technique of weight-sharing, we introduce a new method, FedEx, to accelerate federated hyperparameter tuning that is applicable to widely-used federated optimization methods such as FedAvg and recent variants. Theoretically, we show that a FedEx variant correctly tunes the on-device learning rate in the setting of online convex optimization across devices. Empirically, we show that FedEx can outperform natural baselines for federated hyperparameter tuning by several percentage points on the Shakespeare, FEMNIST, and CIFAR-10 benchmarks, obtaining higher accuracy using the same training budget. 

   more » « less
4. Sparse Bayesian Learning for Sequential Inference of Network Connectivity From Small Data

   https://doi.org/10.1109/TNSE.2024.3471852  

   Wan, Jinming; Kataoka, Jun; Sivakumar, Jayanth; Peña, Eric; Che, Yiming; Sayama, Hiroki; Cheng, Changqing (January 2024, IEEE Transactions on Network Science and Engineering)

   While significant efforts have been attempted in the design, control, and optimization of complex networks, most existing works assume the network structure is known or readily available. However, the network topology can be radically recast after an adversarial attack and may remain unknown for subsequent analysis. In this work, we propose a novel Bayesian sequential learning approach to reconstruct network connectivity adaptively: A sparse Spike and Slab prior is placed on connectivity for all edges, and the connectivity learned from reconstructed nodes will be used to select the next node and update the prior knowledge. Central to our approach is that most realistic networks are sparse, in that the connectivity degree of each node is much smaller compared to the number of nodes in the network. Sequential selection of the most informative nodes is realized via the between-node expected improvement. We corroborate this sequential Bayesian approach in connectivity recovery for a synthetic ultimatum game network and the IEEE-118 power grid system. Results indicate that only a fraction (∼50%) of the nodes need to be interrogated to reveal the network topology. 

   more » « less
5. Scalable Score Computation for Learning Multinomial Bayesian Networks over Distributed Data

   Rao, P.; Katib, A.; Barnard, K.; Kamhaoua, C.; Kwiat, K.; Njilla, L. (November 2016, The AAAI-17 Workshop on Distributed Machine Learning)

   In this paper, we focus on the problem of learning a Bayesian network over distributed data stored in a commodity cluster. Specifically, we address the challenge of computing the scoring function over distributed data in a scalable manner, which is a fundamental task during learning. We propose a novel approach designed to achieve: (a) scalable score computation using the principle of gossiping; (b) lower resource consumption via a probabilistic approach for maintaining scores using the properties of a Markov chain; and (c) effective distribution of tasks during score computation (on large datasets) by synergistically combining well-known hashing techniques. Through theoretical analysis, we show that our approach is superior to a MapReduce-style computation in terms of communication and width. Further, it is superior to the batchstyle processing of MapReduce for recomputing scores when new data are available. 

   more » « less