More Like this

1. Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks

   Morteza Ramezani, Weilin Cong (April 2022, International Conference on Learning (ICLR))

   Despite the recent success of Graph Neural Networks (GNNs), training GNNs on large graphs remains challenging. The limited resource capacities of the existing servers, the dependency between nodes in a graph, and the privacy concern due to the centralized storage and model learning have spurred the need to design an effective distributed algorithm for GNN training. However, existing distributed GNN training methods impose either excessive communication costs or large memory overheads that hinders their scalability. To overcome these issues, we propose a communication-efficient distributed GNN training technique named (LLCG). To reduce the communication and memory overhead, each local machine in LLCG first trains a GNN on its local data by ignoring the dependency between nodes among different machines, then sends the locally trained model to the server for periodic model averaging. However, ignoring node dependency could result in significant performance degradation. To solve the performance degradation, we propose to apply on the server to refine the locally learned models. We rigorously analyze the convergence of distributed methods with periodic model averaging for training GNNs and show that naively applying periodic model averaging but ignoring the dependency between nodes will suffer from an irreducible residual error. However, this residual error can be eliminated by utilizing the proposed global corrections to entail fast convergence rate. Extensive experiments on real-world datasets show that LLCG can significantly improve the efficiency without hurting the performance. One-sentence Summary: We propose LLCG a communication efficient distributed algorithm for training GNNs. 

   more » « less
2. Preliminary Analysis of a Graph-Neural Network Approach to Transmission Expansion Planning

   Xia, J; Birchfield, A B (February 2026, IEEE)

   This paper proposes a scalable method to use Graph Neural Networks (GNNs) to predict contingency analysis overloading for use in transmission expansion planning (TEP). Instead of performing full contingency analysis for candidate transmission lines, the proposed approach employs a trained GNN to quickly predict overload levels under ac power flow contingencies. This preserves the accuracy of ac modeling while delivering exceptional computational speed, enabling evaluation of significantly more candidate lines, making the approach practical for large power systems. The trained GNN is used in a greedy optimization algorithm to determine which candidates lines to add to produce an optimal transmission expansion plan. Results and validation indicate that the proposed method delivers fast, robust transmission plans, highlighting its strong potential for scalability and for addressing larger, more complex transmission planning problems. 

   more » « less
3. SCV-GNN: Sparse Compressed Vector-based Graph Neural Network Aggregation

   https://doi.org/10.1109/TCAD.2023.3291672  

   Unnikrishnan, Nanda K.; Gould, Joe; Parhi, Keshab K. (July 2023, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems)

   Graph neural networks (GNNs) have emerged as a powerful tool to process graph-based data in fields like communication networks, molecular interactions, chemistry, social networks, and neuroscience. GNNs are characterized by the ultra-sparse nature of their adjacency matrix that necessitates the development of dedicated hardware beyond general-purpose sparse matrix multipliers. While there has been extensive research on designing dedicated hardware accelerators for GNNs, few have extensively explored the impact of the sparse storage format on the efficiency of the GNN accelerators. This paper proposes SCV-GNN with the novel sparse compressed vectors (SCV) format optimized for the aggregation operation. We use Z-Morton ordering to derive a data-locality-based computation ordering and partitioning scheme. The paper also presents how the proposed SCV-GNN is scalable on a vector processing system. Experimental results over various datasets show that the proposed method achieves a geometric mean speedup of 7.96× and 7.04× over CSC and CSR aggregation operations, respectively. The proposed method also reduces the memory traffic by a factor of 3.29× and 4.37× over compressed sparse column (CSC) and compressed sparse row (CSR), respectively. Thus, the proposed novel aggregation format reduces the latency and memory access for GNN inference. 

   more » « less
4. Redundancy-Free Computation for Graph Neural Networks

   https://doi.org/10.1145/3394486.3403142  

   Jia, Z.; Lin, S.; Ying, R.; You, J.; Leskovec, J.; Aiken, A. (October 2020, Knowledge discovery and data mining letters)

   Graph Neural Networks (GNNs) are based on repeated aggregations of information from nodes’ neighbors in a graph. However, because nodes share many neighbors, a naive implementation leads to repeated and inefficient aggregations and represents significant computational overhead. Here we propose Hierarchically Aggregated computation Graphs (HAGs), a new GNN representation technique that explicitly avoids redundancy by managing intermediate aggregation results hierarchically and eliminates repeated computations and unnecessary data transfers in GNN training and inference. HAGs perform the same computations and give the same models/accuracy as traditional GNNs, but in a much shorter time due to optimized computations. To identify redundant computations, we introduce an accurate cost function and use a novel search algorithm to find optimized HAGs. Experiments show that the HAG representation significantly outperforms the standard GNN by increasing the end-to-end training throughput by up to 2.8× and reducing the aggregations and data transfers in GNN training by up to 6.3× and 5.6×, with only 0.1% memory overhead. Overall, our results represent an important advancement in speeding-up and scaling-up GNNs without any loss in model predictive performance. 

   more » « less
5. Data Pruning-enabled High Performance and Reliable Graph Neural Network Training on ReRAM-based Processing-in-Memory Accelerators

   https://doi.org/10.1145/3656171  

   Ogbogu, Chukwufumnanya; Joardar, Biresh; Chakrabarty, Krishnendu; Doppa, Jana; Pande, Partha Pratim (September 2024, ACM Transactions on Design Automation of Electronic Systems)

   Graph Neural Networks (GNNs) have achieved remarkable accuracy in cognitive tasks such as predictive analytics on graph-structured data. Hence, they have become very popular in diverse real-world applications. However, GNN training with large real-world graph datasets in edge-computing scenarios is both memory- and compute-intensive. Traditional computing platforms such as CPUs and GPUs do not provide the energy efficiency and low latency required in edge intelligence applications due to their limited memory bandwidth. Resistive random-access memory (ReRAM)-based processing-in-memory (PIM) architectures have been proposed as suitable candidates for accelerating AI applications at the edge, including GNN training. However, ReRAM-based PIM architectures suffer from low reliability due to their limited endurance, and low performance when they are used for GNN training in real-world scenarios with large graphs. In this work, we propose a learning-for-data-pruning framework, which leverages a trained Binary Graph Classifier (BGC) to reduce the size of the input data graph by pruning subgraphs early in the training process to accelerate the GNN training process on ReRAM-based architectures. The proposed light-weight BGC model reduces the amount of redundant information in input graph(s) to speed up the overall training process, improves the reliability of the ReRAM-based PIM accelerator, and reduces the overall training cost. This enables fast, energy-efficient, and reliable GNN training on ReRAM-based architectures. Our experimental results demonstrate that using this learning for data pruning framework, we can accelerate GNN training and improve the reliability of ReRAM-based PIM architectures by up to 1.6×, and reduce the overall training cost by 100× compared to state-of-the-art data pruning techniques. 

   more » « less