N EURAL Networks (NNs) have been used in results such as [1] to approximate the receding-horizon (RH) regulator in RH optimal control. More recently, results in [2] and [3] leverage Deep Neural Networks (DNNs) to approximate the MPC laws. However, the NN approximations in [1]- [4] can only provide statistical guarantees of the approximation error. DNN methods can also be used to capture complex features of the dynamics by using back-propagation algorithms that indicate how to update the inner-layer weights [5]. In results such as [5] and [6], the emergence of DNN models with more complex structures improve function approximation performance. Although DNN function approximation methods show improved performance empirically, these methods typically lack performance guarantees because the accuracy of the outputs are probabilistic. As a result, DNN-based methods may have limited adoption for safety-critical applications.
Motivated to ensure performance guarantees, early works in [7]- [10] use Lyapunov-based methods for NN-based adaptive control of unknown nonlinear systems. In [7]- [9], NNs are trained with a gradient descent-based adaptive update law and used as a feedforward control term. Since the update laws are derived from a stability analysis and the NN weights are embedded inside activation functions, it is challenging to derive adaptation laws from a stability analysis beyond a single-hidden-layer.
In [11] and [12], the authors developed a data-driven adaptive learning method called concurrent learning to increase performance of parameter estimation. Concurrent learning leverages recorded input and output data concurrent to realtime execution to apply batch-like updates to adaptive update laws, and has been extended to works in [13] and [14]. Results in [15], [16], and [17] leverage concurrent learning to develop a deep Model Reference Adaptive Control (D-MRAC). Specifically, a gradient descent-based adaptive update law is used to estimate the ideal output-layer weights of a DNN in real-time online, and an offline data-driven method is used to apply batch updates to the inner-layer weights of the DNN for linear systems with known system matrices. The methods were tested on quadrotors and demonstrated that DNNbased adaptive control can significantly improve learning performance [16], [17]. The authors demonstrated that DNN enabled MRAC outperforms shallow NN MRAC, and also showed that the DNN weights cluster in different regions in different operating envelopes of the quadrotor, clearly establishing the learning performance of DNNs [17]. However, the D-MRAC development is specific to linear systems with known system A, B matrices with matched system uncertainty (x(t)), i.e., ẋ(t) = Ax(t) + B(u(t) + (x(t))).
Building on the output-layer weight adjustment strategy in [15] and [16], this letter develops new control design and stability analysis methods for general uncertain nonlinear systems. A Lyapunov-based adaptive control law is developed to estimate the unknown output-layer weights of the DNN using real-time state feedback. Concurrent to real-time execution, data is collected and an offline function approximation method is used to update the estimates of the inner-layer DNN weights. Moreover, this letter considers control-affine dynamics with uncertain state-dependent control effectiveness matrices. To compensate for the uncertain control effectiveness, a novel adaptive update law is developed that has internal feedback. Specifically, the adaptive update law depends on the control input, and hence, is a function of both the input uncertainty estimates and the DNN weight estimates. To account for switching from iterative updates of the DNN weights, a nonsmooth Lyapunov-based analysis is performed to ensure asymptotic tracking of the desired trajectory. The proposed DNN architecture is shown in Figure 1.
Consider a control-affine nonlinear dynamic system modeled as
where x : [t 0 , ∞) → R n denotes the generalized state, t 0 ∈ R ≥0 denotes the initial time, f : R n → R n denotes the unknown drift dynamics, g : R n → R n×m denotes the uncertain control effectiveness matrix, and u : [t 0 , ∞) → R m denotes the control input. To facilitate the control development, the following assumption is made. Assumption 1: The product of the uncertain control effectiveness matrix and control input can be linearly parameterized as
where Y : R n × R m × [t 0 , ∞) → R n×q denotes a measurable regression matrix, and θ ∈ R q denotes a vector of constant unknown parameters.
Let ⊂ R n be a compact simply connected set with x(t) ∈ , and define S n ( ) as the space where f (x(t)) is continuous. There exists ideal weights, ideal basis functions, and an ideal pre-trained DNN such that the drift dynamics f (x(t)) ∈ S n ( ) can be represented as [18] f (x(t)) = W * T σ * * (x(t)) + ε(x(t)), (3) where W * ∈ R L×n is an unknown bounded ideal output-layer weight matrix, σ * : R p → R L is an unknown bounded vector of the ideal activation functions, * :R n → R p is the ideal unknown DNN, ε : R n → R n is the bounded unknown function reconstruction error associated with the ideal weights, activation functions, and DNN. The ideal unknown DNN * can be expressed as
where k ∈ Z denotes the number of inner-layers of the DNN, the symbol • denotes function composition, W and φ(•) denote the corresponding inner-layer weights and activation functions of the DNN, respectively.
The DNN is updated using a multiple timescale approach. The DNN is trained a priori using data sets collected from previous experiments, simulation data, etc. Ideally, large data sets from the same dynamic system operating under the same environmental conditions will be available for training the DNN. However, the developed strategy of real-time (Lyapunov-based) adjustment of the output-layer weights provides an advantage of significant flexibility in the training data. For example, as observed in the subsequent simulation, the training data could be from a dynamic system with different parameters (i.e., transfer learning), or could also be initialized with random weights.
Based on (3), the DNN approximation of the drift dynamics fi : R n → R n can be represented as fi (x(t)) = W T (t) σi ( i (x(t))), where W :[t 0 , ∞) → R L×n is the estimate of the ideal output-layer weight matrix, σi : R p → R L and i : R n → R p are the i th activation functions and estimates of σ * and * , respectively, and i ∈ N is the DNN estimate update index. 1 The mismatch between the ideal output-layer weights and the weight estimates W: [t 0 , ∞) → R L×n is defined as
Assumption 2: Using the universal function approximation property there exists known constants W * , σ * , σ , ε ∈ R >0 such that the unknown ideal weights W * , unknown ideal activation functions σ * (•), user-selected activation functions σi (•), the unknown ideal DNN * (•), and the function reconstruction error ε(•) can be upper bounded such that sup
A priori training provides 1 (•) and W(t 0 ). The offline DNN training can be achieved by using different techniques. For example, [15] and [16] use a Stochastic Gradient Descent (SGD) based generative network architecture to generate estimates of matched system uncertainty, and the SGD update policy depends on a stochastic estimation of the expected value of the gradient of the loss function over a training set. When the offline DNN training phase is completed, an adaptive control law will be implemented for the system described in (1) to generate the output-layer DNN weight estimates, i.e., W(t) for all t ≥ t 0 . Simultaneous to the online execution, data is collected and offline function approximation methods are used to update estimates on the inner-layer DNN weights.
The control objective is to ensure the trajectory of the system in (1) tracks a desired sufficiently smooth time-varying trajectory x d : [t 0 , ∞) → R n . To quantify the tracking objective, a tracking error e : [t 0 , ∞) → R n is defined as
(5)
To facilitate the subsequent control development, the product of the estimated control effectiveness matrix and the control input can be written as
where ĝ : R n → R n×m denotes the estimate of the control effectiveness matrix. The parameter estimation error θ : [t 0 , ∞) → R q is defined as
where θ : [t 0 , ∞) → R q denotes the parameter estimate. Assumption 3: The estimate of the control effectiveness matrix ĝ is a full-row rank matrix for t ≥ t 0 , and the right pseudo inverse of ĝ(•) is denoted by ĝ+ :R n → R m×n , where
Based on the subsequent stability analysis, the control input is designed as
where k, k s ∈ R >0 are constant control gains, and sgn(•) denotes the signum function. The weight estimate adaptation law is designed as
where W ∈ R L×L denotes a user-defined positive definite, diagonal control gain matrix. The adaptation law for the parameter estimate in ( 6) is designed as
where θ ∈ R q×q denotes a user-defined positive definite, diagonal control gain matrix. Taking the time-derivative of ( 5)
and substituting in ( 1)-( 3) and ( 6)-( 8) yields the closed-loop error system
Recall the initially trained DNN provides initial estimates 1 (•) and W(t 0 ). During implementation of the real-time controller, the output-layer weights of the DNN are estimated online. Concurrently, the data generated in real-time is stored for additional DNN training. Once a sufficient amount of data (user-defined) is collected to improve the function approximation performance, the inner-layer weights of the DNN will be updated to generate i+1 (•) for all i, i.e., ( 8)- (10).
The stability of the DNN-based adaptive tracking controller is established in the following theorem.
Theorem 1: Consider a general nonlinear system modeled by the dynamics in (1) with x(t 0 ) ∈ and satisfying Assumptions 1-3. The control input in (8), the output-layer weight adaptation law in (9), and the parameter estimate adaptation law in (10) ensure the trajectory tracking error defined in (5) yields semi-global asymptotic tracking in the sense that lim t→∞ e(t) → 0, t ≥ t 0 , provided the following gain condition is satisfied
Proof: Consider the candidate Lyapunov-like function
where z : [t 0 , ∞) → R n(L+1)+q is defined as z [ e T , θ T , vec( W) T ] T and vec(•) denotes the vectorization operator. Let ζ : [t 0 , ∞) → R n(L+1)+q be a Filippov solution to the differential inclusion ζ ∈ K[h](ζ ), where ζ(t) = z(t), the calculus of K[ • ] is used to compute Filippov's differential inclusion as defined in [19], and h:R n(L+1)+q → R n(L+1)+q is defined as
The time-derivative of V L exists almost everywhere (a.e.), i.e., for almost all t ∈ [0, ∞), VL (ζ )
where ∂V L (ζ (t)) denotes the Clarke generalized gradient of
where ∇ denotes the gradient operator.
Taking the generalized time-derivative of ( 13), then substituting in the mismatch between the ideal output-layer weight and the weight estimate in (4), the output-layer adaptation law in (9), the parameter estimate adaptation law in (10), and the closed-loop error system in (11) yields
Using the trace operator property, 3 the estimated mismatch for the ideal output-layer weight in (4), and adding and subtracting e T W * T K[ σi ( i (x))] in ( 14
By satisfying the gain condition described in ( 12), ( 16) can be further upper bounded as
Using ( 13) and ( 17) implies 4), ( 5) and ( 7) implies
Using Assumptions 2 and 3 implies σi
Furthermore, by the extension of the LaSalle-Yoshizawa theorem for non-smooth systems in [21] and [22], k e 2 → 0, which implies e(t) → 0 as t → ∞ and x(t) ∈ for all t ≥ t 0 .
To demonstrate the effectiveness of the developed method, a simulation is performed on a control-affine realization of a two-state Van der Pol oscillator. The dynamics used in the simulation are
where The DNN used in this simulation was composed of 4 layers, each with 10, 5, 8, and 2 neurons, respectively. The DNN architecture is illustrated in Figure 2. Each layer is linear and the first, second, and third layers have tangent-sigmoid, logarithmic-sigmoid, and tangent-sigmoid activation functions, respectively. The learning rate (i.e., the learning gain parameter used to determine the step size in retraining the DNN weights at each iteration) was fixed as η = 0.001. The mean squared error (MSE) was used as the loss function for training. Each training iteration lasted until the MSE (i.e., the loss) was less than 10 -3 . The Levenberg-Marquardt algorithm was used to train the weights of the DNN. For each DNN training iteration, 70% of the data was used for training, 15% was used for validation, and 15% was used for testing. To pre-train the DNN, a 600 second simulation of a system with dynamics in (18) and μ = 10 was performed. Training statistics for the offline training are shown in Figure 3. The real-time controller and the update laws in ( 8)- (10) are used to update their respective parametric estimates. Concurrent to the real-time controller execution, input-output data is collected to retrain the DNN. As shown in Figures 456, the training start time is denoted by the red dashed vertical line and the training completion time is denoted by the black dashed vertical line. The time (and corresponding amount of data) between retraining the inner-layer weights is a user-defined parameter of the simulation. After the prescribed time between retraining elapses, the inner-layer DNN weights begin updating via retraining. In this simulation, the time between retraining is 25 seconds. The first retraining starts at t = 25 seconds and ends at t = 37.4 seconds. The second retraining starts at t = 62.4 seconds and ends at t = 68.3 seconds. While the retraining is in process, the real-time controller and update laws continue uninterrupted as described in ( 8)- (10). Once the retraining is completed, the new inner-layer DNN weights are updated, overwriting the previous values. As shown in Figures 456, the time for the MSE to be less than 10 -3 took 12.4 seconds to complete. After the DNN has completed retraining, the controller implements the inner-layer weights at t = 37.4 seconds. After implementing the updated DNN weights, new data is collected for another 25 seconds. To further improve the DNN estimate, a second retraining is performed. During the second training iteration, data from the first 25 seconds and the second 25 seconds are both used. The second retraining took 5.9 seconds. The inner-layer weights from the second retraining are implemented at t = 68.3 seconds.
The tracking error performance in Figure 4 indicates that discretely retrained DNNs with an online adaptive output-layer weights are a viable method to perform trajectory tracking. The first iteration of the DNN (DNN1) is the offline generated DNN, DNN2 is the model after the first retraining, and DNN3 is the model after the second retraining. As shown by the root mean squared error (RMSE) in Table I T . The decrease in error after each retraining is expected since a larger set of system data was used to train the DNN during each retraining. Figure 7 shows the phase plot of the system, and compares the performance of the tracking during the application of each DNN. DNN1 has the worst estimate of the system dynamics. DNN2 and DNN3 show significantly better tracking behavior, which is also reflected in Figure 4. Figure 6 presents the control input to the system for the duration of the simulation. DNN1 poorly approximates the dynamics near x = [-3.5, 3.5] T , and this error is further reflected in Figure 6 with the spikes in control input approximately at t = 2 seconds and t = 8 seconds.
To further demonstrate the flexibility of the developed realtime Lyapunov-based adjustment of the output-layer weights, two additional simulations were performed. In this section, transfer learning-based and randomly initialized DNN weights simulations were investigated. Transfer learning in this context is applying the learned DNN model of one system to another system. In the simulation, transfer learning is demonstrated by training a DNN model on a dataset of a system described by the dynamics in (18) with parameter μ = 1, whereas the simulated system has parameter μ = 10.
In the transfer learning-based approach, the DNN is pretrained with 600 seconds of simulated data from a system with dynamics in (18), but parametrized with μ = 1. Figures 8(a), 9(a), and Table I (B) show the tracking error, phase plot, and RMSE of the transfer learning approach over three iterations of DNN training, respectively. For situations where data cannot be collected a priori, initial inner-layer DNN weights can be selected by the user. In the third simulation, instead of pre-training the DNN, a simulation was performed with the initial DNN randomly selected weights. The performance of the transfer learning-based approach and initial randomly selected DNN weights simulations are depicted in Figures 8 and9. Iterations in the inner-layer weights are shown to improve performance. The first simulation, which was trained with 600 seconds of offline data using the dynamics in (18) with μ = 10 has the best performance with respect to the smallest RMSE within 25 second intervals compared to transfer learning and initial randomly selected DNN weights. Nevertheless, the proposed realtime Lyapunov-based adjustment of the output-layer weights accommodates for different methods to initialize the DNN inner-layer weights.
A multiple timescale DNN-based adaptive control architecture is developed for general nonlinear dynamical systems with unknown drift dynamics and uncertain control effectiveness matrix. Specifically, a Lyapunov-based adaptive update law is developed to estimate the unknown output-layer weights of the DNN and the uncertain control effectiveness matrix in real-time. Simultaneous to real-time execution, data is collected and offline function approximation methods are used to update estimates of the innerlayer weights. A nonsmooth Lyapunov-based analysis is performed to ensure semi-global asymptotic tracking of the desired trajectory. Numerical simulation examples are provided to demonstrate the performance of the proposed architecture.
The subscript i on σ and represents the i th training iteration activation functions and estimated DNN, respectively. The explicit expression for i (x(t)) can be expressed asi (x(t)) = ( W T i,k φi,k • W T i,k-1 φi,k-1 • • • • • W T i,1 φi,1 )(x(t)), where W and φ(•) denote the corresponding estimated innerlayer weights and activation functions of the corresponding training iteration, respectively.
For common activation functions, e.g., hyperbolic tangent function, sigmoid function, radial basis function, σ * = σ = L.
Authorized licensed use limited to: University of Florida. Downloaded on June 25,2021 at 02:44:49 UTC from IEEE Xplore. Restrictions apply.
For real column matrices a, b ∈ R n , the trace of the outer product is equivalent to the inner product, i.e., tr(ba T ) = a T b.