More Like this

1. HePCo: Data-Free Heterogeneous Prompt Consolidation for Continual Federated Learning

   Halbe, Shaunak; Smith, James S; Tian, Junjiao; Kira, Zsolt (December 2023, NeurIPS Workshops - R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Foundation Models)

   In this paper, we focus on the important yet understudied problem of Continual Federated Learning (CFL), where a server communicates with a set of clients to incrementally learn new concepts over time without sharing or storing any data. The complexity of this problem is compounded by challenges from both the Continual and Federated Learning perspectives. Specifically, models trained in a CFL setup suffer from catastrophic forgetting which is exacerbated by data heterogeneity across clients. Existing attempts at this problem tend to impose large overheads on clients and communication channels or require access to stored data which renders them unsuitable for real-world use due to privacy. We study this problem in the context of Foundation Models and showcase their effectiveness in mitigating forgetting while minimizing overhead costs and without requiring access to any stored data. We achieve this by leveraging a prompting based approach (such that only prompts and classifier heads have to be communicated) and proposing a novel and lightweight generation and distillation scheme to aggregate client models at the server. We formulate this problem for image classification and establish strong baselines for comparison, conduct experiments on CIFAR-100 as well as challenging, large-scale datasets like ImageNet-R and DomainNet. Our approach outperforms both existing methods and our own baselines by more than 7% while significantly reducing communication and client-level computation costs. 

   more » « less
2. Quantifying Individual Fairness in Continual Federated Learning

   https://doi.org/10.1145/3701716.3717646  

   Noor, Naima; Mashhadi, Afra (May 2025, ACM)

   Continual Federated Learning (CFL) is a distributed machine learning technique that enables multiple clients to collaboratively train a shared model without sharing their data, while also adapting to new classes without forgetting previously learned ones. This dynamic, adaptive learning process parallels the concept of founda- tion models in FL, where large, pre-trained models are fine-tuned in a decentralized, federated setting. While foundation models in FL leverage pre-trained knowledge as a starting point, CFL continu- ously updates the shared model as new tasks and data distributions emerge, requiring ongoing adaptation. Currently, there are limited evaluation models and metrics in measuring fairness in CFL, and ensuring fairness over time can be challenging as the system evolves. To address this challenge, this article explores temporal fairness in CFL, examining how the fairness of the model can be influenced by the selection and participation of clients over time. Based on individual fairness, we introduce a novel fairness metric that captures temporal aspects of client behavior and evaluates different client selection strategies for their impact on promoting fairness. 

   more » « less
3. Don't Memorize; Mimic The Past: Federated Class Incremental Learning Without Episodic Memory

   Babakniya, Sara; Fabian, Zalan; He, Chaoyang; Soltanolkotabi, Mahdi; Avestimehr, Salman (December 2023, Proceedings of Neural Information Processing Systems)

   Deep learning models are prone to forgetting information learned in the past when trained on new data. This problem becomes even more pronounced in the context of federated learning (FL), where data is decentralized and subject to independent changes for each user. Continual Learning (CL) studies this so-called \textit{catastrophic forgetting} phenomenon primarily in centralized settings, where the learner has direct access to the complete training dataset. However, applying CL techniques to FL is not straightforward due to privacy concerns and resource limitations. This paper presents a framework for federated class incremental learning that utilizes a generative model to synthesize samples from past distributions instead of storing part of past data. Then, clients can leverage the generative model to mitigate catastrophic forgetting locally. The generative model is trained on the server using data-free methods at the end of each task without requesting data from clients. Therefore, it reduces the risk of data leakage as opposed to training it on the client's private data. We demonstrate significant improvements for the CIFAR-100 dataset compared to existing baselines. 

   more » « less
4. PS-FedGAN: An Efficient Federated Learning Framework with Strong Data Privacy

   https://doi.org/10.1109/JIOT.2024.3399226  

   Wijesinghe, Achintha; Zhang, Songyang; Ding, Zhi (January 2024, IEEE Internet of Things Journal)

   Federated Learning (FL) has emerged as an effective paradigm for distributed learning systems owing to its strong potential in exploiting underlying data characteristics while preserving data privacy. In cases of practical data heterogeneity among FL clients in many Internet-of-Things (IoT) applications over wireless networks, however, existing FL frameworks still face challenges in capturing the overall feature properties of local client data that often exhibit disparate distributions. One approach is to apply generative adversarial networks (GANs) in FL to address data heterogeneity by integrating GANs to regenerate anonymous training data without exposing original client data to possible eavesdropping. Despite some successes, existing GAN-based FL frameworks still incur high communication costs and elicit other privacy concerns, limiting their practical applications. To this end, this work proposes a novel FL framework that only applies partial GAN model sharing. This new PS-FedGAN framework effectively addresses heterogeneous data distributions across clients and strengthens privacy preservation at reduced communication costs, especially over wireless networks. Our analysis demonstrates the convergence and privacy benefits of the proposed PS-FEdGAN framework. Through experimental results based on several well-known benchmark datasets, our proposed PS-FedGAN demonstrates strong potential to tackle FL under heterogeneous (non-IID) client data distributions, while improving data privacy and lowering communication overhead. 

   more » « less
5. Benchmarking Algorithms for Federated Domain Generalization

   Bai, Ruqi; Bagchi, Saurabh; Inouye, David I (May 2024, International Conference on Learning Representations (ICLR))

   While prior federated learning (FL) methods mainly consider client heterogeneity, we focus on the Federated Domain Generalization (DG) task, which introduces train-test heterogeneity in the FL context. Existing evaluations in this field are limited in terms of the scale of the clients and dataset diversity. Thus, we propose a Federated DG benchmark that aim to test the limits of current methods with high client heterogeneity, large numbers of clients, and diverse datasets. Towards this objective, we introduce a novel data partition method that allows us to distribute any domain dataset among few or many clients while controlling client heterogeneity. We then introduce and apply our methodology to evaluate 14 DG methods, which include centralized DG methods adapted to the FL context, FL methods that handle client heterogeneity, and methods designed specifically for Federated DG on 7 datasets. Our results suggest that, despite some progress, significant performance gaps remain in Federated DG, especially when evaluating with a large number of clients, high client heterogeneity, or more realistic datasets. Furthermore, our extendable benchmark code will be publicly released to aid in benchmarking future Federated DG approaches. 

   more » « less