More Like this

1. Bayesian Knowledge Distillation: A Bayesian Perspective of Distillation with Uncertainty Quantification

   Fang, Luyang; Chen, Yongkai; Zhong, Wenxuan; Ma, Ping (July 2024, Forty-first International Conference on Machine Learning.)
2. Craft Distillation: Layer-wise Convolutional Neural Network Distillation

   https://doi.org/10.1109/CSCloud-EdgeCom49738.2020.00051  

   Blakeney, Cody; Li, Xiaomin; Yan, Yan; Zong, Ziliang (August 2020, IEEE International Conference on Edge Computing and Scalable Cloud)
3. Fair Graph Distillation

   Feng, Qizhang; Jiang, Zhimeng; Li, Ruiquan; Wang, Yicheng Wang; Zou, Na Zou; Bian, Jiang; Hu, Xia (November 2023, Advances in neural information processing systems)
4. Dual Policy Distillation

   https://doi.org/10.24963/ijcai.2020/435  

   Lai, Kwei-Herng; Zha, Daochen; Li, Yuening; Hu, Xia (July 2020, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence)

   Policy distillation, which transfers a teacher policy to a student policy has achieved great success in challenging tasks of deep reinforcement learning. This teacher-student framework requires a well-trained teacher model which is computationally expensive. Moreover, the performance of the student model could be limited by the teacher model if the teacher model is not optimal. In the light of collaborative learning, we study the feasibility of involving joint intellectual efforts from diverse perspectives of student models. In this work, we introduce dual policy distillation (DPD), a student-student framework in which two learners operate on the same environment to explore different perspectives of the environment and extract knowledge from each other to enhance their learning. The key challenge in developing this dual learning framework is to identify the beneficial knowledge from the peer learner for contemporary learning-based reinforcement learning algorithms, since it is unclear whether the knowledge distilled from an imperfect and noisy peer learner would be helpful. To address the challenge, we theoretically justify that distilling knowledge from a peer learner will lead to policy improvement and propose a disadvantageous distillation strategy based on the theoretical results. The conducted experiments on several continuous control tasks show that the proposed framework achieves superior performance with a learning-based agent and function approximation without the use of expensive teacher models. 

   more » « less
5. Embarrassingly Simple Dataset Distillation

   Feng, Yunzhen; Vedantam, Shanmukha; Kempe, Julia (May 2024, International Conference on Learning Representations)

   Dataset distillation extracts a small set of synthetic training samples from a large dataset with the goal of achieving competitive performance on test data when trained on this sample. In this work, we tackle dataset distillation at its core by treating it directly as a bilevel optimization problem. Re-examining the foundational back-propagation through time method, we study the pronounced variance in the gradients, computational burden, and long-term dependencies. We introduce an improved method: Random Truncated Backpropagation Through Time (RaT-BPTT) to address them. RaT-BPTT incorporates a truncation coupled with a random window, effectively stabilizing the gradients and speeding up the optimization while covering long dependencies. This allows us to establish new state-of-the-art for a variety of standard dataset benchmarks. A deeper dive into the nature of distilled data unveils pronounced intercorrelation. In particular, subsets of distilled datasets tend to exhibit much worse performance than directly distilled smaller datasets of the same size. Leveraging RaT-BPTT, we devise a boosting mechanism that generates distilled datasets that contain subsets with near optimal performance across different data budgets. 

   more » « less