An official website of the United States government
Recently, considerable research attention has been paid to graph embedding, a popular approach to construct representations of vertices in latent space. Due to the curse of dimensionality and sparsity in graphical datasets, this approach has become indispensable for machine learning tasks over large networks. The majority of the existing literature has considered this technique under the assumption that the network is static. However, networks in many applications, including social networks, collaboration networks, and recommender systems, nodes, and edges accrue to a growing network as streaming. A small number of very recent results have addressed the problem of embedding for dynamic networks. However, they either rely on knowledge of vertex attributes, su er high-time complexity or need to be re-trained without closed-form expression. Thus the approach of adapting the existing methods designed for static networks or dynamic networks to the streaming environment faces non-trivial technical challenges. These challenges motivate developing new approaches to the problems of streaming graph embedding. In this paper, we propose a new framework that is able to generate latent representations for new vertices with high e ciency and low complexity under speci ed iteration rounds. We formulate a constrained optimiza- tion problem for the modi cation of the representation resulting from a stream arrival. We show this problem has no closed-form solution and instead develop an online approximation solution. Our solution follows three steps: (1) identify vertices a ected by newly arrived ones, (2) generating latent features for new vertices, and (3) updating the latent features of the most a ected vertices. The new representations are guaranteed to be feasible in the original constrained optimization problem. Meanwhile, the solution only brings about a small change to existing representations and only slightly changes the value of the objective function. Multi-class clas- si cation and clustering on ve real-world networks demonstrate that our model can e ciently update vertex representations and simultaneously achieve comparable or even better performance compared with model retraining.
more »
« less
Backpropagation Computation for Training Graph Attention Networks
Graph Neural Networks (GNNs) are a form of deep learning that have found use for a variety of problems, including the modeling of drug interactions, time-series analysis, and traffic prediction. They represent the problem using non-Euclidian graphs, allowing for a high degree of versatility, and are able to learn complex relationships by iteratively aggregating more contextual information from neighbors that are farther away. Inspired by its power in transformers, Graph Attention Networks (GATs) incorporate an attention mechanism on top of graph aggregation. GATs are considered the state of the art due to their superior performance. To learn the best parameters for a given graph problem, GATs use traditional backpropagation to compute weight updates. To the best of our knowledge, these updates are calculated in software, and closed-form equations describing their calculation for GATs aren’t well known. This paper derives closed-form equations for backpropagation in GATs using matrix notation. These equations can form the basis for design of hardware accelerators for training GATs.
more »
« less
- Award ID(s):
- 1954749
- PAR ID:
- 10470677
- Publisher / Repository:
- Springer
- Date Published:
- Journal Name:
- Journal of Signal Processing Systems
- ISSN:
- 1939-8018
- Subject(s) / Keyword(s):
- Graph attention networks, backpropagation, training
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Inverse kinematics solves the problem of how to control robot arm joints to achieve desired end effector positions, which is critical to any robot arm design and implemen- tations of control algorithms. It is a common misunderstanding that closed-form inverse kinematics analysis is solved. Popular software and algorithms, such as gradient descent or any multi-variant equations solving algorithm, claims solving inverse kinematics but only on the numerical level. While the numerical inverse kinematics solutions are rela- tively straightforward to obtain, these methods often fail, due to dependency on specific numerical values, even when the inverse kinematics solutions exist. Therefore, closed-form inverse kinematics analysis is superior, but there is no generalized automated algorithm. Up till now, the high-level logical reasoning involved in solving closed-form inverse kine- matics made it hard to automate, so it’s handled by human experts. We developed IKBT, a knowledge-based intelligent system that can mimic human experts’ behaviors in solving closed-from inverse kinematics using Behavior Tree. Knowledge and rules used by engineers when solving closed-from inverse kinematics are encoded as actions in Behavior Tree. The order of applying these rules is governed by higher level composite nodes, which resembles the logical reasoning process of engineers. It is also the first time that the dependency of joint variables, an important issue in inverse kinematics analysis, is automatically tracked in graph form. Besides generating closed-form solutions, IKBT also explains its solving strategies in human (engineers) interpretable form. This is a proof-of-concept of using Behavior Trees to solve high-cognitive problems.more » « less
-
null (Ed.)Graph synthesis is a long-standing research problem. Many deep neural networks that learn about latent characteristics of graphs and generate fake graphs have been proposed. However, in many cases their scalability is too high to be used to synthesize large graphs. Recently, one work proposed an interesting scalable idea to learn and generate random walks that can be merged into a graph. Due to its difficulty, however, the random walk-based graph synthesis failed to show state-of-the-art performance in many cases. We present an improved random walk-based method by using negative random walks. In our experiments with 6 datasets and 8 baseline methods, our method shows the best performance in almost all cases. We achieve both high scalability and generation quality.more » « less
-
The authors consider a bistatic configuration with a stationary transmitter transmitting unknown waveforms of opportunity and a single moving receiver and present a deep learning (DL) framework for passive synthetic aperture radar (SAR) imaging. They approach DL from an optimisation based perspective and formulate image reconstruction as a machine learning task. By unfolding the iterations of a proximal gradient descent algorithm, they construct a deep recurrent neural network (RNN) that is parameterised by the transmitted waveforms. They cascade the RNN structure with a decoder stage to form a recurrent auto-encoder architecture. They then use backpropagation to learn transmitted waveforms by training the network in an unsupervised manner using SAR measurements. The highly non-convex problem of backpropagation is guided to a feasible solution over the parameter space by initialising the network with the known components of the SAR forward model. Moreover, prior information regarding the waveform structure is incorporated during initialisation and backpropagation. They demonstrate the effectiveness of the DL-based approach through numerical simulations that show focused, high contrast imagery using a single receiver antenna at realistic signal-to-noise-ratio levels.more » « less
-
Many service systems provide customers with information about the system so that customers can make an informed decision about whether to join or not. Many of these systems provide information in the form of an update. Thus, the information about the system is updated periodically in increments of size [Formula: see text]. It is known that these updates can cause oscillations in the resulting dynamics. However, it is an open problem to explicitly characterize the size of these oscillations when they occur. In this paper, we solve this open problem and show how to exactly calculate the amplitude of these oscillations via a fixed point equation. We also calculate closed form approximations via Taylor expansions of the fixed point equation and show that these approximations are very accurate, especially when [Formula: see text] is large. Our analysis provides new insight for systems that use updates as a way of disseminating information to customers.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1007/s11265-023-01897-1
