An official website of the United States government
Despite the recent popularity of attention-based neural architectures in core AI fields like natural language processing (NLP) and computer vision (CV), their potential in modeling complex physical systems remains underexplored. Learning problems in physical systems are often characterized as discovering operators that map between function spaces based on a few instances of function pairs. This task frequently presents a severely ill-posed PDE inverse problem. In this work, we propose a novel neural operator architecture based on the attention mechanism, which we refer to as the Nonlocal Attention Operator (NAO), and explore its capability in developing a foundation physical model. In particular, we show that the attention mechanism is equivalent to a double integral operator that enables nonlocal interactions among spatial tokens, with a data-dependent kernel characterizing the inverse mapping from data to the hidden parameter field of the underlying operator. As such, the attention mechanism extracts global prior information from training data generated by multiple systems, and suggests the exploratory space in the form of a nonlinear kernel map. Consequently, NAO can address ill-posedness and rank deficiency in inverse PDE problems by encoding regularization and achieving generalizability. We empirically demonstrate the advantages of NAO over baseline neural models in terms of generalizability to unseen data resolutions and system states. Our work not only suggests a novel neural operator architecture for learning interpretable foundation models of physical systems, but also offers a new perspective towards understanding the attention mechanism. Our code and data accompanying this paper are available at https://github.com/fishmoon1234/NAO.
more »
« less
Transformer Meets Boundary Value Inverse Problem
A Transformer-based deep direct sampling method is proposed for electrical impedance tomography, a well-known severely ill-posed nonlinear boundary value inverse problem. A real-time reconstruction is achieved by evaluating the learned inverse operator between carefully designed data and the reconstructed images. An effort is made to give a specific example to a fundamental question: whether and how one can benefit from the theoretical structure of a mathematical problem to develop task-oriented and structure-conforming deep neural networks? Specifically, inspired by direct sampling methods for inverse problems, the 1D boundary data in different frequencies are preprocessed by a partial differential equation-based feature map to yield 2D harmonic extensions as different input channels. Then, by introducing learnable non-local kernels, the direct sampling is recast to a modified attention mechanism. The new method achieves superior accuracy over its predecessors and contemporary operator learners and shows robustness to noises in benchmarks. This research shall strengthen the insights that, despite being invented for natural language processing tasks, the attention mechanism offers great flexibility to be modified in conformity with the a priori mathematical knowledge, which ultimately leads to the design of more physics-compatible neural architectures.
more »
« less
- PAR ID:
- 10423077
- Publisher / Repository:
- ICLR
- Date Published:
- Journal Name:
- International Conference on Learning Representations
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract In this work, we investigate the diffusive optical tomography (DOT) problem in the case that limited boundary measurements are available. Motivated by the direct sampling method (DSM) proposed in Chow et al. (SIAM J Sci Comput 37(4):A1658–A1684, 2015), we develop a deep direct sampling method (DDSM) to recover the inhomogeneous inclusions buried in a homogeneous background. In this method, we design a convolutional neural network to approximate the index functional that mimics the underling mathematical structure. The benefits of the proposed DDSM include fast and easy implementation, capability of incorporating multiple measurements to attain high-quality reconstruction, and advanced robustness against the noise. Numerical experiments show that the reconstruction accuracy is improved without degrading the efficiency, demonstrating its potential for solving the real-world DOT problems.more » « less
-
This work investigates the electrical impedance tomography problem when only limited boundary measurements are available, which is known to be challenging due to the extreme ill-posedness. Based on the direct sampling method (DSM) introduced in [Y. T. Chow, K. Ito, and J. Zou, Inverse Problems, 30 (2016), 095003], we propose deep direct sampling methods (DDSMs) to locate inhomogeneous inclusions in which two types of deep neural networks (DNNs) are constructed to approximate the index function (functional): fully connected neural networks and convolutional neural networks. The proposed DDSMs are easy to be implemented, capable of incorporating multiple Cauchy data pairs to achieve high-quality reconstruction and highly robust with respect to large noise. Additionally, the implementation of DDSMs adopts offline-online decomposition, which helps to reduce a lot of computational costs and makes DDSMs as efficient as the conventional DSM proposed by Chow, Ito, and Zou. The numerical experiments are presented to demonstrate the efficacy and show the potential benefits of combining DNN with DSM.more » « less
-
Abstract In this paper, we consider the inverse scattering problem associated with an anisotropic medium with a conductive boundary. We will assume that the corresponding far–field pattern is known/measured and we consider two inverse problems. First, we show that the far–field data uniquely determines the boundary coefficient. Next, since it is known that anisotropic coefficients are not uniquely determined by this data we will develop a qualitative method to recover the scatterer. To this end, we study the so–called monotonicity method applied to this inverse shape problem. This method has recently been applied to some inverse scattering problems but this is the first time it has been applied to an anisotropic scatterer. This method allows one to recover the scatterer by considering the eigenvalues of an operator associated with the far–field operator. We present some simple numerical reconstructions to illustrate our theory in two dimensions. For our reconstructions, we need to compute the adjoint of the Herglotz wave function as an operator mapping intoH1of a small ball.more » « less
-
Nguyen, Dinh-Liem; Nguyen, Loc; Nguyen, Thi-Phong (Ed.)This paper is concerned with the numerical solution to the direct and inverse electromagnetic scattering problem for bi-anisotropic periodic structures. The direct problem can be reformulated as an integro-di erential equation. We study the existence and uniqueness of solution to the latter equation and analyze a spectral Galerkin method to solve it. This spectral method is based on a periodization technique which allows us to avoid the evaluation of the quasiperiodic Green's tensor and to use the fast Fourier transform in the numerical implementation of the method. For the inverse problem, we study the orthogonality sampling method to reconstruct the periodic structures from scattering data generated by only two incident fields. The sampling method is fast, simple to implement, regularization free, and very robust against noise in the data. Numerical examples for both direct and inverse problems are presented to examine the efficiency of the numerical solvers.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
