More Like this

1. Federated Learning with Matched Averaging

   Wang, Hongyi; Yurochkin, Mikhail; Sun, Yuekai; Papailiopoulos, Dimitris; Khazaeni, Yasaman (April 2020, International Conference on Learning Representations)

   Federated learning allows edge devices to collaboratively learn a shared model while keeping the training data on device, decoupling the ability to do model training from the need to store the data in the cloud. We propose the Federated matched averaging (FedMA) algorithm designed for federated learning of modern neural network architectures e.g. convolutional neural networks (CNNs) and LSTMs. FedMA constructs the shared global model in a layer-wise manner by matching and averaging hidden elements (i.e. channels for convolution layers; hidden states for LSTM; neurons for fully connected layers) with similar feature extraction signatures. Our experiments indicate that FedMA not only outperforms popular state-of-the-art federated learning algorithms on deep CNN and LSTM architectures trained on real world datasets, but also reduces the overall communication burden. 

   more » « less
2. Improving deep learning model performance under parametric constraints for materials informatics applications

   https://doi.org/10.1038/s41598-023-36336-5  

   Gupta, Vishu; Peltekian, Alec; Liao, Wei-keng; Choudhary, Alok; Agrawal, Ankit (December 2023, Scientific Reports)

   Abstract Modern machine learning (ML) and deep learning (DL) techniques using high-dimensional data representations have helped accelerate the materials discovery process by efficiently detecting hidden patterns in existing datasets and linking input representations to output properties for a better understanding of the scientific phenomenon. While a deep neural network comprised of fully connected layers has been widely used for materials property prediction, simply creating a deeper model with a large number of layers often faces with vanishing gradient problem, causing a degradation in the performance, thereby limiting usage. In this paper, we study and propose architectural principles to address the question of improving the performance of model training and inference under fixed parametric constraints. Here, we present a general deep-learning framework based on branched residual learning (BRNet) with fully connected layers that can work with any numerical vector-based representation as input to build accurate models to predict materials properties. We perform model training for materials properties using numerical vectors representing different composition-based attributes of the respective materials and compare the performance of the proposed models against traditional ML and existing DL architectures. We find that the proposed models are significantly more accurate than the ML/DL models for all data sizes by using different composition-based attributes as input. Further, branched learning requires fewer parameters and results in faster model training due to better convergence during the training phase than existing neural networks, thereby efficiently building accurate models for predicting materials properties. 

   more » « less
3. Bridging Data Gaps: A Federated Learning Approach to Heat Emission Prediction in Laser Powder Bed Fusion

   https://doi.org/10.1115/1.4065888  

   Lei, Rong; Guo, Y B; Yan, Jiwang; Guo, Weihong “Grace” (October 2024, Journal of Manufacturing Science and Engineering)

   Abstract Deep learning has impacted defect prediction in additive manufacturing (AM), which is important to ensure process stability and part quality. However, its success depends on extensive training, requiring large, homogeneous datasets—remaining a challenge for the AM industry, particularly for small- and medium-sized enterprises (SMEs). The unique and varied characteristics of AM parts, along with the limited resources of SMEs, hamper data collection, posing difficulties in the independent training of deep learning models. Addressing these concerns requires enabling knowledge sharing from the similarities in the physics of the AM process and defect formation mechanisms while carefully handling privacy concerns. Federated learning (FL) offers a solution to allow collaborative model training across multiple entities without sharing local data. This article introduces an FL framework to predict section-wise heat emission during laser powder bed fusion (LPBF), a vital process signature. It incorporates a customized long short-term memory (LSTM) model for each client, capturing the dynamic AM process's time-series properties without sharing sensitive information. Three advanced FL algorithms are integrated—federated averaging (FedAvg), FedProx, and FedAvgM—to aggregate model weights rather than raw datasets. Experiments demonstrate that the FL framework ensures convergence and maintains prediction performance comparable to individually trained models. This work demonstrates the potential of FL-enabled AM modeling and prediction where SMEs can improve their product quality without compromising data privacy. 

   more » « less
4. How To Backdoor Federated Learning

   Bagdasaryan, Eugene; Veit, Andreas; Hua, Yiqing; Estrin, Deborah; Shmatikov, Vitaly (January 2019, ArXivorg)

   null (Ed.)

   Federated learning enables thousands of participants to construct a deep learning model without sharing their private training data with each other. For example, multiple smartphones can jointly train a next-word predictor for keyboards without revealing what individual users type. We demonstrate that any participant in federated learning can introduce hidden backdoor functionality into the joint global model, e.g., to ensure that an image classifier assigns an attacker-chosen label to images with certain features, or that a word predictor completes certain sentences with an attacker-chosen word. We design and evaluate a new model-poisoning methodology based on model replacement. An attacker selected in a single round of federated learning can cause the global model to immediately reach 100% accuracy on the backdoor task. We evaluate the attack under different assumptions for the standard federated-learning tasks and show that it greatly outperforms data poisoning. Our generic constrain-and-scale technique also evades anomaly detection-based defenses by incorporating the evasion into the attacker's loss function during training. 

   more » « less
5. Few-shot Time-Series Forecasting with Application for Vehicular Traffic Flow

   https://doi.org/10.1109/IRI54793.2022.00018  

   Tran, Victor; Panangadan, Anand (August 2022, Information Reuse and Integration for Data Science (IRI), 2022 IEEE 23rd International Conference on)

   Few-shot machine learning attempts to predict outputs given only a very small number of training examples. The key idea behind most few-shot learning approaches is to pre-train the model with a large number of instances from a different but related class of data, classes for which a large number of instances are available for training. Few-shot learning has been most successfully demonstrated for classification problems using Siamese deep learning neural networks. Few-shot learning is less extensively applied to time-series forecasting. Few-shot forecasting is the task of predicting future values of a time-series even when only a small set of historic time-series is available. Few-shot forecasting has applications in domains where a long history of data is not available. This work describes deep neural network architectures for few-shot forecasting. All the architectures use a Siamese twin network approach to learn a difference function between pairs of time-series, rather than directly forecasting based on historical data as seen in traditional forecasting models. The networks are built using Long short-term memory units (LSTM). During forecasting, a model is able to forecast time-series types that were never seen in the training data by using the few available instances of the new time-series type as reference inputs. The proposed architectures are evaluated on Vehicular traffic data collected in California from the Caltrans Performance Measurement System (PeMS). The models were trained with traffic flow data collected at specific locations and then are evaluated by predicting traffic at different locations at different time horizons (0 to 12 hours). The Mean Absolute Error (MAE) was used as the evaluation metric and also as the loss function for training. The proposed architectures show lower prediction error than a baseline nearest neighbor forecast model. The prediction error increases at longer time horizons. 

   more » « less