More Like this

1. Scaling Up Bayesian Neural Networks with Neural Networks

   Moslemi, Z; Meng, Y; Lan, S; Shahbaba, B (October 2024, Transactions on Machine Learning Research)
2. How neural networks extrapolate: from feedforward to graph neural networks

   Xu, Keyulu; Zhang, Mozhi; Li, Jingling; Du, Simon S.; Kawarabayashi, Ken-ichi; Jegelka, Stefanie (January 2021, International Conference on Learning Representations (ICLR))

   null (Ed.)

   We study how neural networks trained by gradient descent extrapolate, i.e., what they learn outside the support of the training distribution. Previous works report mixed empirical results when extrapolating with neural networks: while feedforward neural networks, a.k.a. multilayer perceptrons (MLPs), do not extrapolate well in certain simple tasks, Graph Neural Networks (GNNs) – structured networks with MLP modules – have shown some success in more complex tasks. Working towards a theoretical explanation, we identify conditions under which MLPs and GNNs extrapolate well. First, we quantify the observation that ReLU MLPs quickly converge to linear functions along any direction from the origin, which implies that ReLU MLPs do not extrapolate most nonlinear functions. But, they can provably learn a linear target function when the training distribution is sufficiently “diverse”. Second, in connection to analyzing the successes and limitations of GNNs, these results suggest a hypothesis for which we provide theoretical and empirical evidence: the success of GNNs in extrapolating algorithmic tasks to new data (e.g., larger graphs or edge weights) relies on encoding task-specific non-linearities in the architecture or features. Our theoretical analysis builds on a connection of over-parameterized networks to the neural tangent kernel. Empirically, our theory holds across different training settings. 

   more » « less
3. Graphon Neural Networks and the Transferability of Graph Neural Networks

   L. Ruiz, L. F. (December 2020, 34th Conference on Neural Information Processing Systems)
4. How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks

   Xu, K.; Zhang, M; Li, J; Du, S.; Kawarabayashi, K; Jegelka, S. (January 2021, International Conference on Learning Representations (ICLR))

   We study how neural networks trained by gradient descent extrapolate, i.e., what they learn outside the support of the training distribution. Previous works report mixed empirical results when extrapolating with neural networks: while feedforward neural networks, a.k.a. multilayer perceptrons (MLPs), do not extrapolate well in certain simple tasks, Graph Neural Networks (GNNs) – structured networks with MLP modules – have shown some success in more complex tasks. Working towards a theoretical explanation, we identify conditions under which MLPs and GNNs extrapolate well. First, we quantify the observation that ReLU MLPs quickly converge to linear functions along any direction from the origin, which implies that ReLU MLPs do not extrapolate most nonlinear functions. But, they can provably learn a linear target function when the training distribution is sufficiently “diverse”. Second, in connection to analyzing the successes and limitations of GNNs, these results suggest a hypothesis for which we provide theoretical and empirical evidence: the success of GNNs in extrapolating algorithmic tasks to new data (e.g., larger graphs or edge weights) relies on encoding task-specific non-linearities in the architecture or features. Our theoretical analysis builds on a connection of over-parameterized networks to the neural tangent kernel. Empirically, our theory holds across different training settings. 

   more » « less
5. coVariance Neural Networks

   Sihag, Saurabh; Mateos, Gonzalo; McMillan, Corey; Ribeiro, Alejandro (October 2022, Advances in neural information processing systems)