This paper considers a class of controllers recently developed using online convex optimization (OCO). Online machine learning and convex optimization methods are powerful tools for learning sequential data. This makes these techniques ideal for high precision control applications like satellite pointing and photolithography. These systems have reliable physics-based models with small error (within the control bandwidth) but are subject to unknown arbitrary disturbances.
This has motivated a large body of recent work using online learning and convex optimization for control [1]- [9]. The most closely related work is the class of OCO controllers defined in [10]. Here, OCO with memory is introduced to the discrete-time control setting as an ideal cost minimization problem (which we describe in detail in Section IV-B) to handle arbitrary disturbances and general time-varying convex cost functions. The OCO controller has promising regret guarantees and makes less restrictive assumptions about the disturbance characteristics (e.g., white noise or worst-case) than that of H 2 and H ∞ optimal control techniques [11], [12]. This makes OCO methods well suited for high precision control applications with unknown, arbitrary disturbances that degrade the system performance.
The OCO framework in [10] aims to learn the disturbance characteristics in real time. However, small model errors can cause instability and thus must be explicitly considered in the design. There are additional works that attempt to learn the model from data [13]- [19]. However, dynamic uncertainties in many high precision applications arise due to high frequency, time-varying, and/or nonlinear effects. It is difficult to learn such unmodeled effects from real-time data. In these cases, it is useful to design a robust OCObased controller that can learn the disturbance features and tolerate model uncertainty, thus motivating our work.
There are three main contributions of our work. First, we provide a robust stability condition for OCO control of a discrete linear time-invariant (LTI) plant (Theorem 1 in Section III-B) and show how it can be used to compute the stability bound (Section V). The robust stability condition is a scaled version of the small gain condition which holds for an arbitrary induced system norm. Second, we show how the robust stability condition can be imposed as a pointwise in time constraint on the OCO controller to ensure robustness to nonparametric uncertainties. This implementation of the robust stability condition is specific to using the induced ℓ ∞norm (Section III-C and Section IV-C), resulting in an easy extension of [10]. Lastly, we present numerical results that illustrate the effect of the robust stability constraint on the OCO controller (Section V).
This section formulates the OCO control problem for discrete-time LTI plants subject to both model uncertainty and unknown disturbances.
Let v ∈ R n be a vector. The p-norm of this vector is defined as ∥v∥ p := n i=1 |v i | p 1 p . Next, N denotes the set of non-negative integers. Let d : N → R n denote a vectorvalued sequence {d 0 , d 1 , . . .}. The ℓ p -norm of d is defined as:
Note that ∥d t ∥ p is the p-norm of the vector d t ∈ R n at time t while ∥d∥ p is the ℓ p -norm of the sequence. The set ℓ p consists of sequences that have finite ℓ p -norm. The subset ℓ pe ¢ ℓ p is the extended space of sequences that have finite ℓ p -norm on all finite intervals, i.e.
T t=0 ∥d t ∥ p p < ∞ for all T g 0. Finally, let G : ℓ pe → ℓ pe denote systems that map an input signal u ∈ ℓ pe to an output signal y ∈ ℓ pe . The induced ℓ p -norm for this system is defined as:
To simplify notation, we'll often use ∥d∥ and ∥G∥ for the signal norm and system induced norm when the specific pnorm is not important.
In this section, we consider the feedback system in Figure 1 and discuss the model uncertainty ∆(z) in more detail. Consider the nominal discrete-time, LTI plant G(z) with dynamics:
where x t ∈ R nx and v t ∈ R nu are the nominal plant state and input at time t, respectively. We assume x 0 = 0 for simplicity.
Model uncertainty for systems with physics-based models often shows up as unmodeled actuator dynamics affecting the plant input [11], [12], [20]. We can account for these unmodeled dynamics by defining an input-multiplicative uncertainty set G ¶ as:
where ¶ ∈ [0, ∞). Note that the induced 2-norm is common choice to bound the uncertainty. However, our main result in Section III holds for any induced p-norm. Let G0 (z) denote the true plant dynamics. We assume that the true plant is within the uncertainty set, i.e. G0 (z) ∈ G ¶ . In other words, there exists a specific ∆ 0 (z) such that ∥∆ 0 ∥ f ¶ and G0 (z) = G(z)(I + ∆ 0 (z)) ∈ G ¶ . More generally, we refer to G(z) = G(z)(I +∆(z)) as the uncertain plant. Here, we assume the uncertainty ∆(z) is LTI. However, our main result in Section III can be extended to the case where ∆ is a possibly nonlinear time-varying (NLTV) system.
Unknown disturbances are often caused by environmental factors and moving physical components which degrade system performance. However, these disturbances often also have learnable characteristics. It is typical to model such disturbances as entering at the plant input as shown in Figures 1 and 2.
OCO control can be used to learn and reject the disturbance without a priori knowledge of the disturbance [1]- [9]. Here, we describe a class of OCO controllers closely related to [10] which considers the case without uncertainty ∆(z) = 0. The OCO controller has the block diagram representation shown in Figure 2. This corresponds to the class of disturbance action controllers defined as:
Fig. 2. Block diagram representation of the OCO controller in a discretetime feedback system with unknown disturbance dt and uncertain plant G(z). The OCO controller is composed of a state-feedback gain K, an estimator E(z), and an LTV system M LTV .
where
t ∈ R nu×nx , and ŵt ∈ R nx are the state-feedback gain, learned coefficients, and disturbance estimate, at time t, respectively. The state-feedback gain K is user-selected while the learned coefficients {M t } H-1 i=0 are typically updated via some online optimization method. For example, [10] uses online projected gradient descent (OPGD) with memory (see Section IV-B).
The online optimization often uses an estimate of the disturbance to learn the coefficients. When there is no uncertainty ∆(z) = 0, the disturbance estimate ŵt can be perfectly estimated from x t and u t using the nominal plant dynamics [10]. We discuss this in more detail in Section IV-A. Thus, we assume the disturbance estimate ŵt to be the output of an LTI estimator E(z) of the following form:
where x e t ∈ R ne and ŵt ∈ R nx are the estimator state and output at time t, respectively. Note that x t and u t are inputs to the estimator.
The first term in ( 5) is considered the baseline controller which we denote by:
The main results in Section III can be generalized to the case when the baseline control u base t is the output of an LTI controller K(z) with input x t . We assume the baseline controller is a static, state-feedback gain for simplicity.
The second term in ( 5) is the output of an finite impulse response (FIR) filter with time-varying coefficients. We denote the FIR filter as a linear time-varying (LTV) system M LTV with input-output dynamics defined as:
where ŵt ∈ R nx and u oco t ∈ R nu are the input and output at time t, respectively. The FIR filter order H is also referred to as the learning horizon since the coefficients are often updated via OCO using the past H disturbance estimates. We provide an example of online optimization in Sections IV and V, but the main results in Section III assume only that the coefficients are time-varying.
The OCO controller (5) can be interpreted as a baseline controller u base t plus an adapting term u oco t which corrects for the unknown disturbance d t based on disturbance estimates.
The uncertainty ∆(z) and disturbance d t have different effects on closed-loop stability. Suppose the state-feedback gain K is stabilizing, i.e., all eigenvalues of (A -BK) are strictly inside the unit disk. Without uncertainty ∆(z) = 0, OCO control can be designed to achieve disturbance rejection with provable guarantees [10]. In this case, a bounded disturbance d cannot cause signals x, u, ŵ, etc. to grow unbounded. However, small amounts of model uncertainty can cause the system to become unstable.
As shown in Figures 1 and 2, the uncertain plant input is the control input perturbed by an unknown disturbance:
where u t , d t , p t ∈ R nu are the control input, disturbance, and perturbed uncertain plant input at time t, respectively. The perturbed input p t is further distorted by the uncertainty ∆(z). The resulting input to the nominal plant G(z) is:
where q t = ∆p t ∈ R nu . Again, v t is the nominal plant input at time t. Not only is there an unknown disturbance d t , but also a distorted signal q t due to the uncertainty ∆(z).
The additional perturbation q t can lead to unexpected behaviors that affect the disturbance estimate and FIR filter coefficient update when left unaccounted for in the OCO design. This can occur even when the state-feedback gain K is stabilizing for the uncertain plant G(z). Thus, the OCO controller is required to: i) learn and compensate for the disturbance, and ii) stabilize the system in the presence of uncertainty. The OCO controller must achieve these objectives without a priori knowledge of the disturbance or uncertainty.
This section provides a condition on M LTV that ensures the feedback system with OCO control remains stable even in the presence of the model uncertainty.
As a first step, we transform the feedback system of the uncertain plant and OCO controller (Figures 1 and 2) to a standard form as shown in Figure 3. This diagram separates the LTI dynamics P from the uncertainty ∆ and time-varying OCO dynamics M LTV . Here P includes the dynamics due to the plant, estimator, and state-feedback gain. This diagram is called a linear fractional transformation (LFT) in the robust control literature [11], [12]. We use the notation F U (P, Γ) for this interconnection with Γ = ∆ 0 0 MLTV closed around the upper channels of P .
Equivalent LFT F U (P, Γ) of original system separating LTI dynamics P from uncertainty ∆ and time-varying learning dynamics M LTV .
An explicit state-space model for P can be determined from the various components of the feedback system described in Section II. The dynamics of P are given by:
Next, we use the LFT representation F U (P, Γ) to formulate and state our robust stability condition.
Our first stability result is a variation of the standard small gain theorem (see Section 5.4 of [21]). This provides a sufficient condition for the dynamics F U (P, Γ) to have a bounded gain from disturbance d to state x. Note stability here is in the sense of bounded gain in some induced norm.
Lemma 1. Consider the interconnection F U (P, Γ) where P : ℓ pe → ℓ pe and Γ : ℓ pe → ℓ pe are linear systems with finite induced ℓ p -norm. Partition P as:
where p := [ p ŵ ] and q := [ q u oco ] are the inputs and outputs of Γ. The interconnection has finite induced ℓ p -norm, i.e. ∥F U (P, Γ)∥ < ∞, if ∥P 11 ∥ ∥Γ∥ < 1.
The system P is LTI so by the principle of superposition (assuming zero initial conditions):
We can bound p using the triangle inequality and the definition of the induced norm:
Next, q = Γp so that ∥q∥ f ∥Γ∥ ∥p∥. Substitute this bound into (13) and re-arrange to obtain:
This last step requires the small gain condition ∥P 11 ∥ ∥Γ∥ < 1 to obtain the bound on ∥p∥.
Finally, the state is x = P 21 q + P 22 d. We can use similar steps and the bound on p to obtain:
Hence, F U (P, Γ) has finite induced ℓ p -norm.
The small gain condition in the previous lemma can be conservative as it does not exploit the block structure Γ = ∆ 0 0 MLTV . We can reduce the conservatism by normalizing the blocks and introducing scalings. Specifically, assume ∥∆∥ f ¶ and ∥M LTV ∥ f ´. Define the normalized uncertainty and learning dynamics as: ∆ = 1 ¶ ∆ and MLTV = 1 ´MLTV . Stacking these together yields
The scaling normalizes each block so that ∥ Γ∥ f 1.
Next, the uncertainty is LTI and hence d 1 ∆ = ∆d 1 for any scalar d 1 > 0. (In fact, this relation holds even if d 1 is also an LTI system but we do not pursue this generalization here.) Similarly, the learning dynamics are also linear and hence d 2 M LTV = M LTV d 2 for any scalar d 2 > 0. It follows that the normalized systems can be equivalently written, for any d 1 , d 2 > 0, as:
This leads to the following scaled small gain result.
Theorem 1. Consider the interconnection F U (P, Γ) where P : ℓ pe → ℓ pe and Γ : ℓ pe → ℓ pe are linear systems with finite induced ℓ p -norm. Assume Γ := ∆ 0 0 MLTV where ∥∆∥ f ¶ and ∥M LTV ∥ f ´. Partition P as:
where p := [ p ŵ ] and q := [ q u oco ] are the inputs and outputs of Γ. The interconnection has finite induced ℓ p -norm, i.e., ∥F U (P, Γ)∥ < ∞, if there exists scalars d 1 , d 2 > 0 such that
satisfies ∥ P11 ∥ < 1.
Proof: Define a scaled version of the nominal dynamics P as:
The constants introduced in the scaled plant P cancel those introduced for Γ in (16). In other words, F U (P, Γ) and F U ( P , Γ) define the same dynamics from d to x. Moreover, ∥ P11 ∥ < 1 and ∥ Γ∥ f 1 by assumption. It follows from the small gain result (Lemma 1) that F U ( P , Γ) = F U (P, Γ) has finite induced ℓ p -norm.
The scalings d 1 and d 2 in the robust stability condition (Theorem 1) can be used to reduce the conservatism of the small gain condition (Lemma 1). They are known as Dscales in the robust control literature (see [22] and Chapter 11 in [11]) and are used in structured singular value robust stability tests. Note that without loss of generality, we can set d 2 = 1 and express (19) in terms of only d 1 . This will be useful in Section V when we use Theorem 1 to compute the stability bound.
In this section, we provide a result specific to the induced ℓ ∞ -norm for the OCO control implementation. The induced ℓ ∞ -norm is useful as it allows us to relate ∥M LTV ∥ ∞→∞ to ∥M t ∥ ∞→∞ . The robust stability constraint can then be imposed as a point-wise in time constraint ´on the coefficients ∥M t ∥ ∞→∞ f ´in the projection step of OPGD. We discuss this further in Section IV-B and IV-C.
The dynamics M LTV in ( 8) can be expressed as:
where
are the stacked FIR coefficients and estimated disturbance history. The following lemma relates the induced ℓ ∞ -norm of the system M LTV to the matrix induced ∞-norm of M t . The proof is based on standard norm properties but is included for completeness.
Lemma 2. Let M LTV be the LTV system defined in (20) and M t be the stacked gains defined in (21). Then
Proof:
The equality in ( 23) is shown in two steps: (A) ∥M LTV ∥ ∞→∞ f sup t ∥M t ∥ ∞→∞ and (B)
First, we show direction (A). Let ŵ and u oco be any inputoutput pair of M LTV . By definition of the induced matrix norm and equation (20), we can show that
It follows that ∥u oco ∥∞ ∥ ŵ∥∞ f sup t ∥M t ∥ ∞→∞ , and thus we have that ∥M LTV ∥ ∞→∞ f sup t ∥M t ∥ ∞→∞ by definition of the induced ℓ ∞ -norm. Hence, claim (A) holds.
Next, we show direction (B). Suppose sup t ∥M t ∥ ∞→∞ achieves its maximum at some finite time t 0 . (The proof can be modified if the supremum occurs as t → ∞.) Then there exists a vector Ŵ * such that ∥ Ŵ * ∥ = 1 and
We can use the vector Ŵ * to construct a signal ŵ * for which: (a) ∥ ŵ * ∥ ∞ = 1, and (b) the corresponding
It follows that ∥u oco ∥∞ ∥ ŵ * ∥∞ g sup t ∥M t ∥ ∞→∞ , and thus we have that ∥M LTV ∥ ∞→∞ g sup t ∥M t ∥ ∞→∞ by definition of the induced ℓ ∞ -norm. Hence, claim (B) holds.
In this section, we demonstrate how the main results can be applied to ensure robust stability of existing OCO controllers. We focus on the OCO controllers in [7], [10] where the coefficients of M LTV are updated via OPGD.
The class of OCO controllers defined by [10] considers the feedback system with OCO control (Figure 2) and no uncertainty (Figure 1) when ∆(z) = 0. In this case, a perfect plant model is assumed G(z) = G(z). Thus, the nominal plant dynamics can be used to design an estimator E(z) and OPGD to update the coefficients in M LTV . Later, we will show how the OPGD projection step can be modified to ensure robust stability for the case that there is uncertainty ∆(z) ̸ = 0.
Without uncertainty, the plant dynamics with unknown disturbance reduce to:
Note that Bd t is the effective disturbance on the state at time t. Assuming the state x t is measurable, we can perfectly reconstruct this effective disturbance at the previous time step. Use the measured state and rearranging the plant dynamics:
With no uncertainty, this estimator perfectly reconstructs the effective disturbance with a one-step delay: ŵt = Bd t-1 . However, perfect reconstruction is no longer guaranteed with uncertainty, i.e. if ∆(z) ̸ = 0 then ŵt ̸ = Bd t-1 . In this case, ŵt is considered an estimate of Bd t-1 .
The disturbance reconstruction (25) can be expressed in state-space form as:
where x e t = -Ax t-1 -Bu t-1 is the estimator state. This has the form of the general LTI estimator E(z) in (6). The estimates ŵt of past disturbances are used to update the FIR coefficients M t defined in (21) by minimizing an "ideal" cost which we describe next.
The coefficients M t are updated at each time step via OPGD in the direction of an "ideal" (per-step) cost. This cost is associated with the nominal plant dynamics (3) and a per-step cost function. Here, we consider quadratic per-step costs:
where
Note that the finite-horizon cost is defined as:
where T is the total time horizon. The ideal cost g(M ) is defined for any static gain M ¢ R nu×nxH based on this perstep cost (26) which is computed and defined as follows.
Let xÄ ∈ R nx and ũÄ ∈ R nu denote the ideal state and control input at time Ä , respectively. The ideal state and input are initialized at Ä = t -H by: xt-H = 0 and ũt-H =
where t is the current time. The ideal state and control input are then computed for Ä = t -H + 1, . . . , t by iterating over the plant dynamics with the static gains M :
The ideal cost is then defined as g(M ) := c(x t , ũt ). In other words, the ideal cost g(M ) is the cost of the plant dynamics evolving with static gain M over the learning horizon H, neglecting dynamics beyond time t -H. The coefficients M t are updated via OPGD on this ideal cost:
where ¸is the learning rate, and Π M is the projection of the gradient step of M t onto a constraint set M. Additional details are given in [7], [10]. Next, we show how the constraint set M can be modified to ensure the robust stability of the OCO feedback system (Figures 1 and 2) when ∆(z) ̸ = 0.
Theorem 1 states that the closed-loop system (Figures 1 and 2) will be stable for bounded uncertainties ∥∆∥ ∞→∞ f ¶ and bounded LTV learning dynamics ∥M LTV ∥ ∞→∞ f ´if the robust stability condition ∥ P11 ∥ ∞→∞ < 1 holds. Larger values of ´risk stability, yet can improve disturbance rejection as they allow the OCO more freedom to adapt the coefficients M t . Thus, it is important to determine the largest possible value of ´such that the robust stability condition holds. We refer to this value of ´as the robust stability bound
´ * which is the solution to:
where P11 is defined in (19). Once the stability bound ´ * is known, we can select the constraint ´to be within [0, ´ * ) to ensure stability. Selecting ´as close as possible to ´ * gives the full benefit of OCO.
Next, it follows from Lemma 2 that if ∥M t ∥ ∞→∞ f
´, then ∥M LTV ∥ ∞→∞ f ´(and vice versa). This means we can simply impose a constraint on the coefficients M t as they are updated via OPGD by defining the constraint set M as:
However, instead of computing the exact projection Π M in (31), we can scale down the coefficients until they are within the constraint set M for simpler online implementation. The projection step (see Algorithm 1 from [10]) can easily be modified to update the coefficients as:
where M step := M t -¸∇M g(M t ) is the gradient step of the coefficients M t at time t. The results in the following section are based on the coefficient update described in (34).
In this section, we perform numerical studies to illustrate the effect of ´and explicitly use the robust stability condition (Theorem 1) to compute the stability bound ´ * . Note that we recover state-feedback when ´= 0 and unconstrained OCO (U-OCO) when ´= ∞. We refer to the case when 0 < ´< ∞ as constrained OCO (C-OCO).
Our example considers the following nominal plant model G(z) and uncertainty ∆(z):
where ∆(z) is small in magnitude at low frequencies and larger at higher frequencies. For the baseline controller, we use a state-feedback gain of K = 0.15. Note that K = 0.15 is stabilizing for both the nominal and uncertain plant. For the OCO, we use the quadratic per-step cost c(x t , u t ) in (26) with Q = 1 and R = 10 -1 , learning rate ¸= 5 × 10 -4 , and learning horizon H = 1. Lastly, we use a step disturbance: d t = 100 for 0 f t f 500 and d t = -100 for 500 < t f T . All simulations were run with a time horizon of T = 1000.
Figure 4 shows the per-step cost c(x t , d t ) and estimated disturbance ŵt of U-OCO at each time t. We compare the performance of the nominal (red dashed) and uncertain (blue solid) plant. The disturbance is perfectly reconstructed ŵt = Bd t-1 (see Section IV-A) with the nominal plant. However, with the uncertain plant, this is not the case ŵt ̸ = Bd t-1 . The ideal cost g(M ) computation assumes the nominal plant model and perfect disturbance estimation. This mismatch introduces an error in the coefficient update M t+1 which causes instability. The instability is illustrated by the perstep cost and estimated disturbance growing unbounded. On the other hand, U-OCO performance is stable without uncertainty because the disturbance is estimated perfectly. Thus, the constraint ´is needed for the coefficient update to ensure stability for the uncertain plant.
Figure 5 shows the per-step cost c(x t , d t ) and estimated disturbance ŵt of C-OCO for ´= 1.5 at each time t. Again, we compare the performance of the nominal (red dashed) and uncertain (blue solid) plant. As mentioned before, an error in the disturbance estimate introduces an error in the ideal cost gradient. The ideal cost gradient error can cause the gradient step M step = M t -∇ M g(M t ) to grow too large in the wrong direction. When the constraint ´is chosen such that the robust stability condition (Theorem 1) is satisfied, the effect of uncertainty induced error on the gradient step of the coefficient update is limited. This is illustrated in Figure 5 as the performance of C-OCO on the uncertain plant eventually recovers the performance on the nominal model with ´= 1.5. Thus, imposing the constraint ´can ensure that OCO is robust to uncertainty. bound also depends on the learning horizon H which is not taken advantage of in the robust stability condition in Theorem 1.
In this paper, we establish a robust stability condition using the small gain theorem for a class of OCO controllers with memory and use the result to compute the robust stability bound. In particular, we impose the stability bound as a constraint on the controller point-wise in time. We provide numerical results to illustrate that imposing the robust stability constraint will ensure stability in the presence of bounded uncertainties. Future work will focus on reducing the conservatism of the robust stability bound ´ * , computing the exact projection, and developing an OCO controller for the output-feedback case.
J. Lai and P. Seiler are with the Department of Electrical Engineering and Computer Science at the University of Michigan, Ann Arbor, MI 48109, USA. {joycelai,pseiler}@umich.edu
Authorized licensed use limited to: University of Michigan Library. Downloaded on June 10,2025 at 14:50:50 UTC from IEEE Xplore. Restrictions apply.