More Like this

1. Large graph property prediction via graph segment training

   Cao, Kaidi; Phothilimthana, Phitchaya Mangpo; Abu-El-Haija, Sami; Zelle, Dustin; Zhou, Yanqi; Mendis, Charith; Leskovec, Jure; Perozzi, Bryan (May 2024, International Conference on Neural Information Processing Systems)
2. Large Graph Property Prediction via Graph Segment Training

   Cao, Kaidi; Phothilimthana, Phitchaya Mangpo; Abu-El-Haija, Sami; Zelle, Dustin; Zhou, Yanqi; Mendis, Charith; Leskovec, Jure; Perozzi, Bryan (December 2023, Advances in neural information processing systems)

   Learning to predict properties of a large graph is challenging because each prediction requires the knowledge of an entire graph, while the amount of memory available during training is bounded. Here we propose Graph Segment Training (GST), a general framework that utilizes a divide-and-conquer approach to allow learning large graph property prediction with a constant memory footprint. GST first divides a large graph into segments and then backpropagates through only a few segments sampled per training iteration. We refine the GST paradigm by introducing a historical embedding table to efficiently obtain embeddings for segments not sampled for backpropagation. To mitigate the staleness of historical embeddings, we design two novel techniques. First, we finetune the prediction head to fix the input distribution shift. Second, we introduce Stale Embedding Dropout to drop some stale embeddings during training to reduce bias. We evaluate our complete method GST+EFD (with all the techniques together) on two large graph property prediction benchmarks: MalNet and TpuGraphs. Our experiments show that GST+EFD is both memory-efficient and fast, while offering a slight boost on test accuracy over a typical full graph training regime. 

   more » « less
3. Generalization Guarantee of Training Graph Convolutional Networks with Graph Topology Sampling

   Li, Hongkang Li; Wang, Meng; Liu, Sijia; Chen, Pin-Yu; Xiong, Jinjun Xiong. (January 2022, 2022 International Conference on Machine Learning (ICML))
4. Graph Coarsening via Convolution Matching for Scalable Graph Neural Network Training

   Dickens, C; Huang, E; Reganti, A; Zhu, J; Subbian, K; Koutra, D (May 2024, TheWebConf’24 Workshop on Data-Centric Artificial Intelligence (DCAI-WebConf’24))
5. Backpropagation Computation for Training Graph Attention Networks

   https://doi.org/10.1007/s11265-023-01897-1  

   Gould, Joe; Parhi, Keshab K. (October 2023, Journal of Signal Processing Systems)

   Graph Neural Networks (GNNs) are a form of deep learning that have found use for a variety of problems, including the modeling of drug interactions, time-series analysis, and traffic prediction. They represent the problem using non-Euclidian graphs, allowing for a high degree of versatility, and are able to learn complex relationships by iteratively aggregating more contextual information from neighbors that are farther away. Inspired by its power in transformers, Graph Attention Networks (GATs) incorporate an attention mechanism on top of graph aggregation. GATs are considered the state of the art due to their superior performance. To learn the best parameters for a given graph problem, GATs use traditional backpropagation to compute weight updates. To the best of our knowledge, these updates are calculated in software, and closed-form equations describing their calculation for GATs aren’t well known. This paper derives closed-form equations for backpropagation in GATs using matrix notation. These equations can form the basis for design of hardware accelerators for training GATs. 

   more » « less