This article is part of the theme issue ‘Frontiers of applied inverse problems in science and engineering’.
]]>Operator learning between in nite-dimensional function spaces is an important task that arises in many disciplines of science and engineering. In recent years, deep neural networks have been successfully applied to learn operators for solving numerical partial di erential equations (PDEs) [1][2][3], image processing [4] and inverse problems [5,6].
Operator learning is challenging in general, since the input and output functions lie in in nitedimensional spaces. To address this di culty, many deep operator learning approaches are proposed in an encoder-decoder framework. This framework employs encoders to map the input and output functions to nite-dimensional vectors and then learn a map between these dimensionreduced vectors. Popular deep operator learning methods in this encoder-decoder framework include PCANet with principal component analysis [1,7,8] and Fourier neural operators (FNOs) based on fast Fourier transforms [9][10][11][12][13]). These methods utilize deterministic or data-driven linear encoders and decoders for dimension reduction, and neural networks are used to learn the map. For nonlinear dimension reduction, autoencoders have demonstrated success in extracting lowdimensional nonlinear structures in data [14][15][16][17], and autoencoders have been applied to operator learning in Seidman et al. [18]; Kontolati et al. [19]; Liu et al. [20]. Other widely used deep operator learning architectures include the Deep Operator Network (DeepONet) [3,[21][22][23][24][25][26][27][28][29]; random feature models [30], among others.
Apart from the computational advances, theoretical works were established to understand the representation and generalization capabilities of deep operator learning methods. A theoretical foundation on the approximation property of PCANet was established in Bha acharya et al. [1]. This was followed by a more comprehensive study in Lanthaler [31], which derived both upper and lower bounds for such approximations. Further contributions were made in [17], where a generalization error bound was established for the encoder-decoder-based neural networks, including PCANet as a speci c instance. The universal approximation property of FNOs was analysed in Kovachki et al. [32]. DeepONets were proposed based on a universal approximation theory in Chen & Chen [33], and were further analysed in Lanthaler et al. [34]; Schwab et al. [35]. More recently, neural scaling laws governing DeepONets were investigated in [36] based on approximation and generalization theories. Lanthaler [37] explored lower bounds on the parameter complexity of operator learning, demonstrating that achieving a power scaling law for general Lipschi operators is theoretically impossible, despite the empirical observations in [3,8]. However, these theoretical frameworks remain limited in fully explaining the empirical success of deep neural networks in operator learning, primarily due to the inherent challenges posed by the curse of dimensionality.
In science and engineering applications, many datasets exhibit low-dimensional structures. For example, even though the images in ImageNet [38] have an ambient dimension exceeding 150 000. Pope et al. [39] showed that the ImageNet dataset has an intrinsic dimension of approximately 40. Many high-dimensional datasets exhibit repetitive pa erns and special structures including rotation and translation, which contribute to a low intrinsic dimensionality [40,41]. In PDE-related inverse problems, one often needs to infer certain unknown parameters in the PDE from given observations, such as inferring the permeability in porous media equations given the solutions [42]. In inverse problems, the unknown parameters often exhibit a low-dimensional structure. For example, the unknown parameters considered in Lu et al. [43] and Hasani & Ward [44] are generated by just a few Fourier bases.
By leveraging low-dimensional structures, existing deep learning theory has shown that the approximation and generalization errors of deep neural networks for function estimation converge at a fast rate depending on the intrinsic dimension of data or learning tasks [45][46][47][48][49], by contrast to a slower rate of convergence in high-dimensional spaces [43,50,51]. A generalization error bound of an autoencoder-based neural network (AENet) for operator learning was established in Liu et al. [20], where the convergence rate depends on the intrinsic dimension of the dataset. Recently, generalization error bounds were established in Dahal et al. [52]; Havrilla & Liao [53] for generative models and transformer neural networks when the input data lie on a low-dimensional manifold. Another approach to mitigate the curse of dimensionality considers operators with special structures, such as those arising in elliptic PDEs [54] or holomorphic operators [55,56].
In this paper, we address operator learning problems arising from inverse problems [11,57] to determine inverse Quantities of Interest (QoIs) associated with PDEs given observations on PDE solutions. In many practical scenarios, the inverse QoIs in PDEs exhibit a simpler structure compared to the corresponding PDE solutions. For instance, in the context of heat di usion in non-uniform materials, the di usivity parameter is inherently material dependent. If the domain can be partitioned into several subregions, each characterized nearly by a uniform material property, the di usivity parameter can be represented by a piecewise constant function [58]. A similar framework is explored in Vaidya et al. [59] for the convection-di usion of solutes in heterogeneous media. When the unknown parameter in PDEs is piecewise constant, it can be expressed by a linear combination of characteristic functions, each corresponding to a distinct subregion. These characteristic functions form a set of orthogonal bases. By contrast, the solutions to PDEs often exhibit more complex structures. Consequently, it is natural to assume that the PDE parameters possess low-dimensional linear structures, while the PDE solutions exhibit low-dimensional nonlinear structures.
Motivated by this observation, we propose a novel framework termed a Coe cient-to-Basis Network (C2BNet) to solve inverse problems. The proposed network architecture comprises two key components: (i) a coe cient network, which maps the input function (representing the PDE solution) to the coe cients corresponding to the basis functions of the output (PDE parameter), and (ii) a basis network (a linear layer), which is responsible for learning the basis functions in the output space. C2BNet is di erent from existing encoder-decoder based networks, such as PCANet [1] and AENet [20]. PCANet uses principle component analysis (PCA) for dimension reduction, which can only discover low-dimensional linear structures, while C2BNet can learn low-dimensional nonlinear structures of PDE solutions. AENet uses an autoencoder to explore low-dimensional nonlinear structures in data. The training of AENet contains two steps: in the rst step, an autoencoder is trained to learn low-dimensional data structures in the input. In the second step, another network is trained to learn a mapping from the input latent variable given by the trained autoencoder to the output. In C2BNet, we do not need to explicitly train an autoencoder for dimension reduction. Both the coe cient network and basis network are trained together, which simpli es the training process.
A signi cant challenge in solving inverse problems by machine learning lies in adapting to new tasks or inferring QoIs in di erent regions of the domain based on new data. A common scenario arises when the new QoIs are discretized on a ner mesh, leading to a higher-dimensional output compared with the previous tasks. To address this challenge, we propose a ne-tuning approach for a pre-trained network that was initially trained on coarser discretizations. Our derivation demonstrates that only updating a linear layer of the pre-trained network is su cient to adapt to the new task on ner discretizations. This approach signi cantly reduces the computational complexity associated with retraining the entire network on the new dataset while maintaining the accuracy of the prediction. By assuming that the input functions reside on a low-dimensional manifold, we establish approximation and generalization error bounds for our proposed C2BNet. In particular, these error bounds depend crucially on the intrinsic dimension of the input functions. Our contributions are summarized as follows:
royalsocietypublishing.org/journal/rsta Phil. Trans. R. Soc. A 383: 20240054 ......................................................................................................................... -We introduce C2BNet, a novel framework for learning operators designed for PDE-based inverse problems. C2BNet can be e ciently ne-tuned to accommodate di erent discretizations using a pre-trained network, signi cantly reducing the computational cost of retraining without sacri cing accuracy. -We establish approximation and generalization error bounds for C2BNet incorporating low-dimensional structures in datasets. Our results demonstrate that C2BNet adapts to low-dimensional data structures without an additional encoder-decoder mechanism. -We validate the performance of C2BNet and the theoretical ndings through comprehensive numerical experiments. The remainder of this paper is structured as follows. In §2, we introduce some concepts and background de nitions necessary for an understanding of the proposed framework. In §3 we present details of the proposed C2BNet. A theoretical analysis of C2BNet is provided in §4, with proofs detailed in appendix A. The e cacy of C2BNet is demonstrated through a series of numerical experiments in §5. Finally, we conclude the paper in §6. Notation: In this paper, we use bold le ers to denote vectors and normal le ers to denote scalars. Calligraphic le ers are used to denote sets. For a set Ω, we use |Ω| to denote its volume.
We rst introduce some de nitions about manifolds. We refer readers to Tu [60] and Lee [61] for more details.
The transformation in a chart de nes a coordinate map on Q. An atlas of M is a collection of charts that covers M:
A nite atlas is an atlas with a nite number of charts.
On a smooth manifold, we de ne C s functions as follows.
De nition 3 (C s functions on M). Let M be a smooth manifold and f ∶ M → ℝ be a function dened on M. The function is called a C s function on M if for any chart (Q, ) of M, the composition
To measure the complexity of a manifold, we de ne reach as follows.
De nition 4 (Reach [62,63]). Let M be a manifold embedded in ℝ D . We de ne
z‖ 2 } , royalsocietypublishing.org/journal/rsta Phil. Trans. R. Soc. A 383: 20240054 ......................................................................................................................... where d(x, M) is the distance from x to M. The reach of M is de ned as = inf x∈M inf y∈G ‖x -y‖ 2 .
The reach of a manifold gives a characterization of curvature. A hyper-plane has a reach = ∞, and a hyper-sphere with radius r has a reach = r.
In this paper, we consider feedforward neural networks in the form of
where W l 's are weight matrices, b l 's are bias and ReLU(a) = max{a, 0} is the recti ed linear unit and is applied elementwise to its argument. The function de ned in equation (2.1) is a network with L layers. The number of layers refers to the number of weight matrices that are used in the network. The width of the l-th layer refers to the number of rows of W l , and the width of the network is the largest number of rows among all weight matrices. We consider the following network class:
gives the number of nonzero elements of its argument.
Our objective is to learn an unknown operator
where X ⊂ L 2 (Ω X ) and Y ⊂ L 2 (Ω Y ) are input and output function sets with domain Ω X ⊂ ℝ s 1 and Ω Y ⊂ ℝ s 2 respectively. In addition, X and Y belong to separable Hilbert spaces with inner products ⟨u 1 , u 2 ⟩ X and ⟨v 1 , v 2 ⟩ Y , respectively. In PDE-based inverse problems, the input set X contains the PDE solutions and the output set Y contains the PDE parameters to be determined. Motivated by low-dimensional linear structures in the PDE parameters [58,59], we assume that the output functions in Y approximately lie in a low-dimensional subspace.
Assumption 1. Let d 2 > 0 be an integer. Suppose there exists a set of orthonormal functions
where denotes the approximation error which may depend on d 2 .
We denote
.2) Assumption 1 ensures the existence of a set of orthonormal bases whose span can approximate functions in Y with error. This assumption is inspired by nite element methods and existing royalsocietypublishing.org/journal/rsta Phil. Trans. R. Soc. A 383: 20240054 .........................................................................................................................
works on operator learning. In nite element methods [64], the output function space is approximated by a nite element space, spanned by a set of basis functions. For the encoder-decoderbased operator learning approaches, such as PCANet and FNOs [1,10,17,31], linear encoders and decoders are used based on certain bases. In these works, the bases are either deterministic, such as Fourier bases and Legendre polynomials, or estimated by data-driven tools, such as PCA.
For any input u ∈ X and output v ∈ Y, we denote their discretized counterparts by u = S X (u) ∈
, where S X and S Y denote the corresponding discretization operators. For a given discretization operator S X , we denote the induced inner product in ℝ D 1 by ⟨⋅, ⋅⟩ S X . One way to de ne such an inner produce is by using a quadrature rule:
where k > 0 are the weights in the quadrature rule. When the functions in X are smooth and the discretization grid of S X is su ciently ne, we expect ‖S X (u)‖ S X ≈ ‖u‖ X for any u ∈ X . We make the following assumption on the discretization operator S Y :
Assumption 2 holds as long as functions in Y are uniformly regular and the discretization grid is su ciently ne. For example, according to the Nyquist-Shannon sampling theorem [65], bandlimited functions can be completely determined from their discretized counterparts on a suciently ne grid. A speci c example is provided in [20, Example 1] to demonstrate that assumption 2 can be satis ed if the functions in Y are band limited and the discretization grid is su ciently ne. For another example, when functions in Y are bounded and second-order di erentiable, assumption 2 can be satis ed by choosing ⟨⋅, ⋅⟩ S Y as the composite trapezoidal rule with su ciently ne grids. We can nd the weight k 's in equation (3.3) using a quadrature rule to approximate the continuous inner product by the discretized counterpart.
which implies that each coe cient is a functional of the input function u as well. We construct a coe cient network
where u = S X (u) denotes the discretized counterpart of u. Given a discretization grid in Ω Y associated with the discretization operator S Y , we can represent the basis functions
. In C2BNet, we use a linear layer to learn this set:
Our operator network Ψ NN that approximates the target operator Ψ in equation (3.1) is constructed as
The architecture of Ψ NN is illustrated in gure 1a. Although both C2BNet can DeepONet have two subnetworks, their structures are di erent. DeepONet approximates an operator through a linear combination of learnable basis functions (trunk) and the learnable coe cients (branch). However, C2BNet is a feedforward network. It is a composition of two networks instead of a product of two networks.
A key advantage C2BNet is adapting to di erent discretizations. Once the network is trained on a speci c discretization associated with S Y , it can be e ciently ne-tuned to accommodate a new discretization associated with S ′ Y . As indicated by equation (3.4), the coe cients
k=1 are determined solely by the input function u and are independent of the discretization S Y . In our framework, altering the discretization in Ω Y only requires updating the basis network
. Consequently, if the coe cient network f coef learns the coe cients well, only f basis needs to be ne-tuned in the new dataset to learn the updated basis functions. This strategy eliminates the need to retrain the entire network, as only a single linear layer needs to be updated, which signi cantly reduces the computational cost. Our ne-tuning strategy is illustrated in gure 1b. In fact, since the ne-tuning of C2BNet only updates the last linear layer, the training loss is quadratic with respect to the weights of the last layer. We can solve it explicitly instead of training a network using stochastic gradient decent or any other algorithms. This property distinguishes C2BNet from other encoder-decoder based networks which need to retrain part or the whole network.
In this section, approximation and generalization error bounds are established for the C2BNet in §3. C2BNet is designed to utilize low-dimensional linear structures in output functions. By further leveraging low-dimensional nonlinear structures in input functions, we prove approximation and generalization error bounds depending on the intrinsic dimension of the input functions. All proofs are deferred until appendix A.
Our theoretical analysis requires some assumptions. The rst assumption below says that the functions in X and Y have a bounded domain and the function values are bounded. Assumption 3. Suppose the followings hold for X and Y: there exist (i) Suppose S X discretizes functions in X without loss of information: there exists a map
be an atlas of M. For any given chart (U k , k ) in this atlas, there exists a constant C such that for any
Assumption 4 assumes that X processes a low-dimensional nonlinear structure. Assumption 4(i) imposes a one-to-one correspondence between each function in X and their discretized counterpart. Assumption 4(i) holds when the functions in X are bandlimited and the discretization grid is su ciently ne. Assumptions 4(ii)-(iii) assume that X exhibits a low-dimensional nonlinear structure.
Our last assumption is about the target operator Ψ:
Assumption 5 imposes a Lipschi property of the target operator Ψ, which is common in the operator learning literature.
We next provide a simple example showing that assumptions 1-5 are satis ed.
Example 1. Consider the transport equation
The objective is to infer f(t) from observed data sampled from u(x, t). Suppose f(x) is a linear function:
), = 0, and assumption 1 is satis ed.
For any given a, b, the PDE solution is
Assumption 3 is satis ed. The operator maps u(x, t) to f(t) = arcsin(u(x, t) -x), assumption 5 is satis ed. We sample N x equally spaced points along x and N t equally spaced points along t.
When N x ≥ 2, N t ≥ 3, the expression of u(x, t) can be uniquely determined from the sampled points.
Assumption 4(i) is satis ed by choosing D X as this mapping. Since both u(x, t) and u are parameterized by (a, b), they are located on a manifold M with d 1 = 2. Assumption 4(ii) is satis ed. Let royalsocietypublishing.org/journal/rsta Phil. Trans. R. Soc. A 383: 20240054 .........................................................................................................................
be the mapping from u to (a, b). Then (M, ) is a chart (and also an atlas) of M. For (a, b) ∈ (M),
, which is Lipschtiz in a and b. Assumption 4(iii) is satis ed. By choosing ⟨⋅, ⋅⟩ S X as the composite trapezoidal rules, assumption 2 is satis ed when N x , N t is su ciently large. In this example, all of assumptions 1-5 are satis ed.
Our rst theoretical result is on the approximation power of C2BNet for Lipschi operators under assumptions 2-5.
and suppose assumptions 2-4 hold. For any > 0, there exists a network architecture
and a linear network
) such that for any operator Ψ satisfying assumption 5, such an architecture gives rise to fcoef ∈ F coef , fbasis ∈ F basis and ˜ = fbasis
The constant hidden in O depends on
and the surface area of M.
Theorem 1 is proved in appendix A.1. Theorem 1 shows that if the network architecture is properly chosen, C2BNet can universally approximate Lipschi operators after a linear projection with an arbitrary accuracy. Furthermore, the size of the network scales with with an exponent depending on the intrinsic dimension d 1 of the input functions , instead of the ambient dimension D 1 .
Our C2BNet in theorem 1 is constructed to approximate Proj•Ψ. When the outputs of Ψ approximately lie in a low-dimensional subspace as assumed in assumption 1, we can combine theorem 1 and assumption 1 to guarantee an approximation error of Ψ.
L , > 0, and suppose that assumptions 1-4 hold. For any > 0, there exists a network architecture
and a linear network
) such that for any operator satisfying assumption 5, such an architecture gives rise to fcoef ∈ F coef , fbasis ∈ F basis and ˜ = fbasis • fcoef with
Suppose we are given input-output pairs S = {u i , vi } n i=1 with u i ∈ X sampled from a probability distribution and vi = Ψ(u i ) + i , where i 's represent independent and identically distributed (i.i.d.) noise satisfying the following assumption: Assumption 6. Suppose i 's are i.i.d. copies of ∈ Y. For any y ∈ Ω Y , i (y) follows a sub-Gaussian distribution with variance proxy 2 .
Given a network architecture F coef and F basis , we de ne the C2BNet class
Given the training data {u i , vi } n i=1 with u i = S X (u i ), vi = S Y (v i ), we train C2BNet through the following empirical risk minimization:
The generalization error of Ψ satis es the following upper bound.
and consider the C2BNet class F de ned in equation (4.5). The minimizer of equation (4.6), denoted by Ψ, satis es
where C is a constant. Both C and the constants hidden in O depend on
and the surface area of M.
Theorem 2 is proved in appendix A.3. The error bound in theorem 2 has two components: the rst is the network estimation error, which converges to zero as n goes to in nity; the second represents the loss of information while output functions are projected to a d 2 -dimensional subspace. If functions in Y lie on a d 2 -dimensional subspace, then = 0 and the squared generalization error in theorem 2 converges at the rate n -2 2+d 1 . This rate of convergence depends on the intrinsic dimension d 1 instead of the ambient dimension D 1 . By solving equation (4.6), C2BNet can adapt to low-dimensional structures in input functions and enjoys a fast convergence rate depending on the intrinsic dimension of the input functions, a enuating the curse of dimensionality. If Y is not exactly lying in a d 2 -dimensional subspace, the projection error usually decays as d 2 increases. We can choose d 2 to be large enough so that is small.
In this section, we demonstrate the e ectiveness of C2BNet through three sets of numerical experiments. To show the e ectiveness and robustness of the methods, we use the same network
In this example, we examine the radiative transfer equation (RTE) [66][67][68] with a contrast sca ering parameter (x), the goal is to infer the inverse QoIs (x) given the solution of the PDE. The RTE is a fundamental model in optical tomography and represents a classic inverse problem. The objective is to recover the sca ering parameter based on given observations. The RTE is described by the following equation:
Here, s is a vector in the unit sphere S n-1 , and n denotes the spatial dimension of the problem.
In our experiments, we set n = 2, making S n-1 = S 1 , which corresponds to the unit circle. In addition, we set = 1 and the domain D = [0, 1] 2 .
To close the model, Dirichlet boundary conditions are imposed. Speci cally, for directions entering the domain (s ⋅ n < 0), the boundary condition is de ned as I(x, s) = I in . This applies to the boundary subset
where n is the unit outward normal vector at x ∈ D. The condition is expressed as I(x, s) = I in (x, s) for all (x, s) ∈ -.
In our examples, the top, bo om and right-hand boundaries are assigned zero incoming boundary conditions, while the left-hand boundary is assumed to have a non-zero ow, injecting energy or particles into the domain.
The term 'contrast sca ering' refers to the di erence in sca ering properties between speci c channels and the surrounding background. Each sca ering includes ve channels, where the size of each is randomly determined by two free variables: length and width. Figure 2 illustrates four realizations of the sca ering parameter , highlighting the variability in channel structures across di erent samples.
In this section, we present the numerical results. For input, the solution (observation) is uniformly discretized on an 11 × 11 spatial mesh, with four velocity directions uniformly distributed on the unit circle. The inverse QoIs correspond to the target sca ering parameter, which is discretized on a 10 × 10 spatial mesh, d low = 50 for this example. In gure 3, we present the error decay in the L 1 norm with respect to the number of training samples. From the left-hand panel of the gure, we observe that the relative error decreases to as low as 3% when 144 training samples are used, demonstrating the e ectiveness of the neural network. Moreover, the right-hand panel shows the error decay on a log scale linearly, validating the power-law behaviour shown in theorem 2.
In this example, we consider the elliptic equation, where (x) is the permeability. The goal is to recover (x, y) given the observations. We set (x, y) = w 1 sin(x + y) + w 2 cos(x + y) + w 3 sin(2(x + y)) + w 4 cos(2(x + y))+4.1, where w i ∼ Uniform(-1, 1). We present two realizations in gure 4.
In this section, we present the numerical results. The input observations (solutions of the equation) are uniformly sampled using a 10 × 10 mesh, while the inverse QoIs are also discretized on a 10 × 10 mesh. The prediction error is measured using the L 2 norm, with the results illustrated in gure 5. From the left-hand panel of gure 5, we observe that the relative L 2 error decreases to 0.7% with 360 training samples and remains below 1% even with only 108 training samples. This demonstrates the accuracy of the network's predictions. Furthermore, as shown in the righthand panel of gure 5, errors exhibit a linear decay in the logarithmic scale, which validates the theoretical results presented in theorem 2.
(ii) Transfer to new QoIs (permeability on a ner mesh)
We test the performance of C2BNet by modifying the QoIs while keeping the encoding part of the trained network xed. This approach signi cantly reduces the computational cost of inferring new QoIs by avoiding the need to train an entirely new network. Speci cally, the new inverse QoIs correspond to solutions de ned on a ner 20 × 20 grid, which di ers from the 10 × 10 training mesh evaluated in §5b(i) To adapt the model, we retain all but the nal layer and train only the last linear layer, which has a dimension of 12 × 400 (d low = 12). The convergence of errors with respect to the number of training samples is presented in gure 6. Left: Error comparison between the proposed method, which transfers the trained weights except for the last layer and updates only the last layer, and the approach of retraining all layers for the downstream task with 20 × 20 inverse QoIs. Notably, with signi cantly less trainable parameters, the proposed method achieves a similar prediction error as the full retraining. Right: error decay in log scale and linear t the log error. For comparison, we also retrain all layers of the network for the new task, where the QoIs are de ned on a 20 × 20 mesh. In particular, the full retraining process requires more cost than the proposed approach, which updates only the nal layer. Speci cally, complete retraining requires updating a total of 36 312 parameters, while the proposed method only updates the last layer, which consists of 4800 parameters. However, the proposed method achieves relative errors similar to full retraining, demonstrating its e ectiveness and accuracy; see gure 7.
In this example, we consider the parabolic equation,
with Dirichlet boundary conditions. The goal is to nd the initial condition given the solution at the terminal time. The initial condition used has the form: u(x, 0) = ∑3 i=1 w i sin(i x) + p i cos(i x), where w i , p i are the weights uniformly sampled from [-1, 1]. We display four initial condition realizations in gure 8. (i) Results for the time-dependent di usion equation The network input (observations) are solutions at t = 0.01, and we take 64 points uniformly from [0, 2], while the inverse QoIs are the initial conditions u(x, 0) discretized by a 64-equal-distance mesh. The results are presented in gure 9. As illustrated in the left-hand panel, the network achieves prediction relative errors below 1% when approximately 225 training samples are used. Furthermore, the error decay is plo ed on a logarithmic scale with respect to the number of training samples. The observed linear decay con rms the power-law behaviour established in the theorem.
(ii) Transfer to new QoIs (solutions on a ner mesh)
To demonstrate that the latent layer (the second-to-last layer) provides valuable information, as established in the theorem, we consider a new set of QoIs corresponding to initial conditions dened on a more dense mesh with 127 grid points. The proposed method updates only the nal linear layer of the network, which has a size of 20 × 127 (d low = 20), while keeping all preceding layers unchanged. This approach signi cantly reduces training costs compared with full retraining. The convergence results are presented in gure 10. From the left-hand panel of the gure, we observe a linear decay when applying a logarithmic scale to both the error and the number of training samples. The right-hand panel illustrates a close match in convergence rates between the pre-trained network, which predicts the original 64-point discretized initial condition, and the new network, where only a single linear layer is retrained for the updated QoIs.
For comparison, we evaluate the proposed method against the approach of retraining all layers for the new tasks. The total number of trainable parameters in the full retraining method is 312,60, approximately 12.31 times larger than the proposed method with only 2540 parameters, which only tunes the last linear layer. Despite signi cant savings in training cost, the accuracy of the proposed method remains unchanged. See gure 11 for the comparison.
In this work, we introduced C2BNet to solve inverse PDE problems within the operator learning paradigm. C2BNet consists of two components, a coe cient network and a linear basis network. C2BNet e ciently adapts to di erent discretizations through ne-tuning, leveraging a pre-trained model to signi cantly reduce computational costs while maintaining high accuracy. We also established approximation and generalization error bounds for learning Lipschi operators using C2BNet. Our theories show that C2BNet is adaptive to low-dimensional data structures Left: Error comparison between the proposed method, which transfers the trained weights except for the last layer and updates only the last layer, and the approach of retraining all layers for the downstream task with 127-dimensional inverse QoIs. Notably, the full retraining involves approximately 12 times more trainable parameters than the proposed method; however, the prediction errors behave similarly. Right: error decay in log scale and linear t the log error.
and achieves a fast convergence rate depending on the intrinsic dimension of the dataset. The efcacy of C2BNet is demonstrated by systematic numerical experiments. The results con rm that C2BNet achieves a strong balance between computational e ciency and accuracy. These ndings highlight the potential of C2BNet as a powerful and scalable tool to solve inverse problems in scienti c computing and engineering applications.
The success of C2BNet relies on low-dimensional data structures. When the data exhibit complex, high-dimensional structures lacking such low-dimensional representations, larger values of d 1 and d 2 must be employed to mitigate information loss. Under these conditions, the inherent advantages of C2BNet may be diminished. Nevertheless, the framework remains applicable and can be e ectively utilized for e cient ne-tuning tasks.
Data accessibility. The data for this manuscript are available at Purdue University Research Repository [69]. Declaration of AI use. We have not used AI-assisted technologies in creating this article. Authors' contributions. Z.Z.: conceptualization, data curation, formal analysis, investigation, methodology, software, validation, visualization, writing-original draft, writing-review and editing; H.L.: conceptualization, data curation, formal analysis, investigation, methodology, software, validation, visualization, writingoriginal draft, writing-review and editing; W.L.: conceptualization, funding acquisition, project administration, supervision, writing-review and editing; G.L.: conceptualization, funding acquisition, project administration, resources, supervision, writing-review and editing.
royalsocietypublishing.org/journal/rsta Phil. Trans. R. Soc. A 383: 20240054 ......................................................................................................................... Now we consider the bases in equation (A 1). Since { k } d 2 k=1 is independent of u, for any given discretization grids S Y , {S Y ( k )} d 2 k=1 is a set of xed vectors, which can be realized by a matrix of dimension D 2 × d 2 . Speci cally, let W
We have
where the last inequality uses equation (A 2) and the fact that the absolute value of every element of W is bounded by M 2 . Se ing 1 = d 2 M 2 nishes the proof. ■
Proof of corollary 1. Let Ψ be the network constructed in theorem 1. According to assumption 1, we have
Proof of theorem 2. We decompose the error as
.
We next derive bounds for T 1 and T 2 .
royalsocietypublishing.org/journal/rsta Phil. Trans. R. Soc. A 383: 20240054 ......................................................................................................................... Bounding T 1 . We bound T 1 as (A 3) For the rst term, we have 2 inf
For any 1 > 0, theorem 1 shows that there exist network architectures F coef = F NN (D 1 , d 2 , L, p, K, , R)
where
Substituting equations (A 6) and (A 7) into equation (A 3) gives rise to function. For any z 1 , z 2 ∈ (U), denote u j = -1 (z j ), u j = D X (u j ), v j = Ψ(u j ) for j = 1, 2. We have
According to assumption 4(iii), D X • -1 is a Lipschi function. For any given nite atlas of M, there exists a constant C M that is an upper bound of the Lipschi constant of D X • -1 for all 's in the atlas. Applying this property to equation (B 1) gives rise to
■
royalsocietypublishing.org/journal/rsta Phil. Trans. R. Soc. A 383: 20240054 .........................................................................................................................