cientific computation and simulation have increasingly used in engineering analysis and design. The primary role of computational models is to make predictions for both analysis and design. Given a set of conditions or a design, we can predict how a component or system behaves. The computational model, however, is never perfect and exact, and uncertainty is always present. For example, the model input may have random variables, and as a result, the model prediction, the model output, or the response, is also uncertain. It is therefore important to quantify the uncertainty associated with the response predicted, and this is the major task of uncertainty quantification (UQ) [1][2][3].
Since uncertainty always exists in model parameters and model predictions, UQ is performed to quantify the effect of uncertainty in model input and model error on the model output (prediction). UQ needs to call computational models repeatedly to obtain probability distributions of responses given distributions of model input variables. In general, UQ is computationally expensive, especially when the dimension of model input is high [4,5]. To this end, regression is commonly used to build a surrogate model to replace the original computational model. Regression can be performed by many methods, such as response surface modeling, Gaussian Process (GP) method, support vector machines (SVM), and neural network [6][7][8]. Regression starts from generating a set of input samples, getting labels (outputs) by calling the computational model at the input samples, and then fitting a surrogate model with the input samples and their labels.
As indicated by many studies, including our own, machine learning is particularly useful for UQ problems encountered in engineering design. Machine learning techniques can significantly reduce the computational burden in the following aspects: Reduce the dimension of uncertain input variables [9] and create accurate but inexpensive surrogate models [10][11][12][13], both for higher computational efficiency. Specific machine learning techniques have increasingly used. For example, the GP method for quantifying model structure uncertainty [14,15]; the SVM method for estimating rare event probabilities [16], and other methods for predicting component and system reliability [17]. Many studies show that add physical constraints can significantly improve prediction accuracy [18,19]. For instance, the constraints from physics can be added to the loss function of artificial neural networks [18], used as a priori information for deep learning [20], and incorporated into GP [21].
Getting labels is usually costly since it runs the computational model many times. Physics-informed neural network (PINN) is proposed [22,23] to obtain the solution of partial differential equations (PDEs) without solving the PDEs by classical numerical methods (e.g. the Finite Element Method). The PDEs are served as the so-called physical constraints during the training process of the neural network. This method has gained much attention because it makes the regression task feasible without solving the true label. Besides, the physical constraints prevent the regression from severe overfitting in conventional neural network, especially when labeled data are limited.
Motivated by the label-free deep learning, this study investigates a label-free UQ method so that the computational efficiency can be enhanced. The method works for problems where responses are solved out from a system of equations derived from physical principles, and solving the system of equations is expensive while evaluating them is inexpensive. We call these equations physical equations. The strategy is to use only the input samples to build a surrogate model that satisfies the physical equations without solving them. As a result, there is no need to generate labels. Neural network is used to build the surrogate model.
Computational models are given by
where 𝒙𝒙 is a vector of input variables, and 𝑦𝑦 𝑖𝑖 is a response. In many applications. The relationship between the input and output is not explicit, and it is derived from the following system of physical equations:
is the solution of the system of physical equations. Solving the system of physical equations is usually computationally expensive.
The task of UQ in this study is illustrated in Fig. 1 and is stated as follows:
• Given: Distributions of 𝒙𝒙 and the system of physical equations ℎ 𝑖𝑖 (𝒙𝒙, 𝑦𝑦 𝑖𝑖 ) = 0, 𝑖𝑖 = 1,2 … 𝑚𝑚
• Find: Distributions of 𝒚𝒚
After the joint distribution of 𝒚𝒚 is available, it can be used for reliability analysis, robustness assessment, and many other analysis tasks, as demonstrated in Example 1. It can also be used for design under uncertainty, such as reliabilitybased design, robust design, and design decision making, as demonstrated in Example 2.
Traditional UQ needs to evaluate computational models 𝑦𝑦 𝑖𝑖 = 𝑓𝑓(𝒙𝒙), where 𝑦𝑦 𝑖𝑖 , 𝑖𝑖 = 1,2, … , 𝑚𝑚, must be solved out from the system of physical equations in Eq. ( 2). Solving for 𝑦𝑦 𝑖𝑖 may be computationally expensive but evaluating the system of physical equations is much cheaper. The label-free UQ method does not solve the system of physical equations and only evaluates them because no labels are needed. The deep learning neural network method is used. The label-free UQ method consists of two steps.
Step 1 creates surrogate models. The surrogate models are given by
The regression method we use is neural network, which builds surrogate models based on data. In regular neural network regression, a dataset with labels is given by ��𝒙𝒙 (𝑖𝑖) , 𝒚𝒚 (𝑖𝑖) �� 𝑖𝑖=1,𝑑𝑑 , where 𝑑𝑑 is the size of the dataset. The model parameters of a neural network, which are to be learned, include weights and biases in all layers. They are denoted by 𝒘𝒘.
The to-be-determined parameters 𝒘𝒘 will be obtained during the model training process. In this work, we do not have labels, and our dataset is ��𝒙𝒙 (𝑖𝑖) �� 𝑖𝑖=1,𝑑𝑑 . To this end, we will use the system of physical equations directly as proposed in other physics-based informed deep learning [22]. The strategy is to satisfy the physical equations at the training points ��𝒙𝒙 (𝑖𝑖) �� 𝑖𝑖=1,𝑑𝑑 .
where
To satisfy the physical equations in Eq. ( 4), we define a loss function
If 𝐿𝐿(𝒘𝒘) = 0, all the physical equations will be satisfied. Minimizing the loss function while training the neural network, we obtain the estimated model parameters 𝒘𝒘; namely 𝒘𝒘 * = arg min 𝐿𝐿(𝒘𝒘) (7) Then surrogate models 𝑦𝑦 𝑖𝑖 = 𝑓𝑓 𝑗𝑗 𝑁𝑁𝑁𝑁 (𝒙𝒙), 𝑖𝑖 = 1,2, … , 𝑚𝑚, are available.
Step 2 performs UQ based on 𝑦𝑦 𝑖𝑖 = 𝑔𝑔 � 𝑖𝑖 (𝒙𝒙). Many existing UQ methods can be used. Since the surrogate models are inexpensive, we may use Monte Carlo simulation (MCS) to find the distribution of a response.
The architecture of the neural network and procedure is illustrated in Fig. 2.
Two examples are used to demonstrate the label-free UQ method. The first example is an analysis problem where the probability distributions of responses are produced. The second example is a design problem where the design requirement is satisfied at a given probability level.
As shown in Fig. 3, a person rows a boat to cross a river from points A to B with a constant velocity 𝑣𝑣 𝑀𝑀 . If the river flows with a velocity 𝑣𝑣 𝑅𝑅 , the task is to determine 𝑣𝑣 𝑀𝑀 and the angle 𝜃𝜃 that the boat should direct. It is the angle between the relative velocity of the boat 𝑣𝑣 𝑀𝑀/𝑅𝑅 with respect to the river and the horizonal direction. The two physical equations are therefore
The random input variables are 𝒙𝒙 = (𝑥𝑥 1 , 𝑥𝑥 2 , 𝑥𝑥 3 ) T = (𝑣𝑣 𝑀𝑀/𝑅𝑅 , 𝑣𝑣 𝑅𝑅 , 𝛽𝛽) T , and their distributions are given in Table 1.
The first step is to generate models 𝑣𝑣 𝑀𝑀 = 𝑓𝑓 1 𝑁𝑁𝑁𝑁 �𝑣𝑣 𝑀𝑀/𝑅𝑅 , 𝑣𝑣 𝑅𝑅 , 𝛽𝛽�, and 𝜃𝜃 = 𝑓𝑓 2 𝑁𝑁𝑁𝑁 �𝑣𝑣 𝑀𝑀/𝑅𝑅 , 𝑣𝑣 𝑅𝑅 , 𝛽𝛽�. 𝑑𝑑 = 200 training points are used. The neural network includes six layers with four hidden layers, one input layer, and one customer regression output layer. Each hidden layer has 20 neutrons. The activation function is a hyperbolic tangent function. The learning rate is 0.01, and the optimizer is Adam.
A scatter plot of the predictions and true labels of the two responses at the training points is given in Fig. 4. The true labels are obtained by solved the two physical equations in Eq. (10). Note that in real applications, the true labels are not available. The scatter plot is for only a demonstration purpose. The scatter plot indicates good accuracy of the surrogate models.
After the two models are built, MCS is used for UQ. The sample size is 10 6 . The PDFs of the two responses are provided in Figs. 5 and6. The distribution parameters, means and standard deviations, of the two responses, are given in Tables 2 and3. For a verification purpose, we also provide the accurate results from true responses and MCS with a sample size 10 6 . The results show the good accuracy of the label-free UQ. Table 3 Statistics of 𝜽𝜽
The design task is to design a cantilever shaft [24,25] as shown in Fig. 7 so that it can withstand random forces 𝐹𝐹 and 𝑃𝑃; and a random torque 𝑇𝑇, with the reliability greater than or equal to [𝑅𝑅] = 0.999. The random variables are 𝒙𝒙 = �𝑆𝑆 𝑦𝑦 , 𝑃𝑃, 𝐹𝐹, 𝑇𝑇� T , where 𝑆𝑆 𝑦𝑦 is the yield strength of the material. All the random variables are independent, and their distributions are given in Table 5. 𝐿𝐿 is the length of the tube, and 𝐿𝐿 = 0.15 m, The design variables are 𝒅𝒅 = (𝑑𝑑 0 , 𝑡𝑡) T , which should be chosen from the following list of preferred sizes for 𝑑𝑑 0 × 𝑡𝑡 (mm): 12×2, 16×2, 16×3, 20×4, 24×4, 25×5, 30×4, 30×5, 42×4, 42×5, 50×4, 50×5. The distributions of random variables are given in Table 4.
The design requirement is that the maximum stress should be less than the yield strength, and the design margin equation is derived from the strength theory, which provides a physical equation at the limit state.
where 𝒙𝒙 = �𝑆𝑆 𝑦𝑦 , 𝑃𝑃, 𝐹𝐹, 𝑇𝑇, 𝑡𝑡� T . 4 ) 64 ( 12)
The diameter 𝑑𝑑 0 is the response; namely, 𝑦𝑦 = 𝑑𝑑 0 . The input variables are 𝒙𝒙 = (𝑥𝑥 1 , 𝑥𝑥 2 , 𝑥𝑥 3 , 𝑥𝑥 4 , 𝑥𝑥 5 ) T = �𝑆𝑆 𝑦𝑦 , 𝑃𝑃, 𝐹𝐹, 𝑇𝑇, 𝑡𝑡� T . Equation 11 is rewritten as
where
We generate 𝑑𝑑 = 200 training points ��𝒙𝒙 (𝑖𝑖) �� 𝑖𝑖=1,200 . We then use the label-free UQ method to build a surrogate model 𝑦𝑦 = 𝑓𝑓 𝑁𝑁𝑁𝑁 (𝒙𝒙), or 𝑑𝑑 0 = 𝑓𝑓 𝑁𝑁𝑁𝑁 (𝑆𝑆 𝑦𝑦 , 𝑃𝑃, 𝐹𝐹, 𝑇𝑇, 𝑡𝑡). The neural network contains three hidden layer, one feature input To verify the accuracy, we generate true labels at training points ��𝒙𝒙 (𝑖𝑖) �� 𝑖𝑖=1,200 . The scatter plot of the model output is provided in Fig. 8. In the range of 𝑑𝑑 0 ∈ [30, 60] mm, the model produces highly accurate predictions. The prediction of the reliability of a design is also accurate if the design falls into this range.
Since the surrogate model efficient, we use Monte Carlo simulation (MCS) to conduct several analyses, from where we can find the best design solution. We select a value of 𝑡𝑡 from the list of the preferred design variables and perform MCS using the distributions given in Table 4. This yields a distribution of 𝑑𝑑 0 , which is the minimum 𝑑𝑑 0 , denoted by 𝑑𝑑 0 𝑚𝑚𝑖𝑖𝑛𝑛 , for the limit state. Since the limit state is determined by random variables, 𝑑𝑑 0 𝑚𝑚𝑖𝑖𝑛𝑛 is also a random variable.
How do we choose a deterministic design variable 𝑑𝑑 0 for the required reliability [𝑅𝑅]? We use the percentile value to find 𝑑𝑑 0 .
We now obtain a pair of design variables (𝑑𝑑 0 , 𝑡𝑡) T for a given value of 𝑡𝑡. We then repeat this process until all values of 𝑡𝑡 are used. The results are given in the first two columns of Table 5. Using the list of referred design variables, the candidate designs are determined, and they are 42×5, 50×4, and 50×5. Considering a smaller size of the shaft, we select the final design to be 42×5. To verify the reliability, we calculate the actual reliability at the calculated diameters with the true function in Eq. ( 17) by MCS. The sample size of MCS is 10 7 . The actual reliability is higher than and close to the required one. Since the final design 42×5 is larger than the calculated design 38.36×5, the final reliability will also be larger than the required reliability. The PDF of the required 𝑑𝑑 0 obtained from the label-free UQ is given in Fig. 9. Note that in real applications, labels are not available, and MCS with real labels is not feasible, Using MCS herein is for only a demonstration purpose. In this example, there are two responses, but with only one physical equation. This is the case where the dimension of outputs is larger than the number of physical equations. This example indicates that this case can also be handled by the proposed method. This example also demonstrates that the label-free UQ method can be used for design. The model built from the label-free method may be used for other tasks, such as optimization under uncertainty and robust design.
This study demonstrates the feasibility of label-free uncertainty quantification using neural network regression. A surrogate model is built without labels. The training points only contain a dataset of input variables at which a set of physical equations are satisfied during the learning process. The satisfaction is achieved by employing a loss function which is defined using a set of physical equations. The two examples demonstrate the potential applications of the method. The future work will be the extension to more general applications, where time-and space-dependent responses and models are involved. The other possible research direction is to integrate the regression and uncertainty quantification so that the quantified uncertainty is part of the regression output.