The classical optimal and robust control problems, including linear quadratic regulator (LQR), linear quadratic Gaussian (LQG) control, and H ∞ control, have been extensively studied (Kalman, 1963;Levine and Athans, 1970). It is well known that almost all of these problems are nonconvex in the space of controller (i.e., policy) parameters. Nevertheless, classical techniques based on controller reparameterizations (Scherer and Weiland, 2015;Boyd et al., 1994) or Riccati equations (Zhou et al., 1996) have been established to characterize optimal or suboptimal controllers. These classical techniques do not optimize over the policy parameters directly, and often require an explicit system model. In contrast, the optimization landscapes of optimal and robust control problems offer a fruitful alternative perspective. In this approach, the control costs are viewed as functions of the policy parameters, and their analytical and geometrical properties are studied (Lewis, 2007;Hu et al., 2023;Talebi et al., 2024). This perspective is naturally amenable to data-driven design paradigms such as reinforcement learning and learning-based control (Recht, 2019).
However, this policy optimization perspective for control generally leads to nonconvex and potentially nonsmooth problems. For example, the set of feedback gains K that stabilize the system ẋ = Ax + Bu via u = Kx is already nonconvex; if we consider output-feedback controller synthesis such as LQG, then the parameterized set of dynamic policies can even be disconnected (Tang et al., 2023). Furthermore, the LQG cost function may have spurious stationary points (Zheng et al., 2022), and there may exist uncountably many globally optimal policies lying on a manifold induced by similarity transformations (Zheng et al., 2022;Tang et al., 2023;Kraisler and Mesbahi, 2024).
In addition to non-convexity, non-smoothness may also arise when considering robust control problems. A typical performance measure for robust control is the H ∞ norm of a certain closed-loop transfer function (Zhou et al., 1996), which is known to be both nonconvex and nonsmooth in the policy space (Apkarian and Noll, 2006;Lewis, 2007).
For nonconvex and nonsmooth optimization, it is generally very hard to establish theoretical guarantees for local search algorithms. Nevertheless, recent findings have revealed benign nonconvex landscape properties in several benchmark control problems, including LQR (Fazel et al., 2018;Mohammadi et al., 2022;Fatkhullin and Polyak, 2021), risk-sensitive control (Zhang et al., 2021), LQG (Tang et al., 2023;Zheng et al., 2022;Duan et al., 2024), dynamic filtering (Umenberger et al., 2022;Zhang et al., 2023), H ∞ control (Hu and Zheng, 2022;Guo and Hu, 2022;Tang and Zheng, 2023), and distributed control (Furieri et al., 2020a). Many of these works leveraged the idea that the control problems under investigation admit suitable convex reformulations. However, these existing works are mostly on a case-by-case basis. Our work aims to provide a unified framework that explains the benign nonconvex landscape properties of these iconic control problems.
We introduce a unified framework, called Extended Convex Lifting (ECL), to reveal hidden convexity in classical optimal and robust control problems from a modern optimization perspective. The core idea behind ECL stems from the existing results in control theory that, via a suitable change of variables, many optimal and robust control problems admit "convex reformulations" (Scherer and Weiland, 2015;Boyd et al., 1994). By fitting the change of variables into the ECL framework, we can analyze the nonconvex landscape of the corresponding policy optimization problem by exploiting its hidden convexity. Figure 1 provides a schematic illustration of ECL. Specifically, we begin with the epigraph of the objective J(K), and lift a certain subset of its closure to a higher-dimensional set L lft by incorporating Lyapunov variables. Then, we construct a smooth bijection Φ that maps L lft to a product space F cvx × G aux where F cvx is convex. This bijection encodes the change of variables associated with the control problem. In many control problems, F cvx can be represented by linear matrix inequalities (LMIs), while G aux accounts for similarity transformations of output-feedback policies.
Despite non-convexity and non-smoothness, the existence of an ECL not only shows that the policy optimization problem is equivalent to a convex problem (Theorem 2.1) but also identifies a class of non-degenerate stationary points that are globally optimal (Theorem 3.1). Our ECL framework covers many iconic control problems; many recent results on the global optimality of (nondegenerate) stationary points, such as LQR (Fazel et al., 2018;Mohammadi et al., 2022), LQG (Tang et al., 2023), state-feedback H ∞ control (Guo and Hu, 2022), output-feedback H ∞ control (Tang and Zheng, 2023), are special cases once the corresponding ECL is constructed.
We point out that our ECL framework is more general than Tang et al. (2023); Umenberger et al. (2022); Guo and Hu (2022); Sun and Fazel (2021); Mohammadi et al. (2022), as it can directly handle both state-feedback and output-feedback policies, as well as smooth and nonsmooth cost functions. More importantly, the ECL framework naturally distinguishes between degenerate and non-degenerate policies, reflecting the subtleties between strict and non-strict LMIs in control. Due to space limitations, we omit most proofs in this paper; detailed proofs, additional discussions, and illustrative examples can be found in our extended reports (Zheng et al., 2023(Zheng et al., , 2024)).
2. Extended Convex Lifting (ECL) for Benign Non-convexity
We first provide a motivating example. Consider an LTI system ẋ(t) = Ax(t)+Bu(t)+w(t) with
, where w(t) is a white Gaussian noise with E[w(t)w(τ )] = 4δ(t-τ )I. We aim to design a state-feedback policy
Standard calculations show that the cost is
Following the classical change of variables Y = KX where X solves the Lyapunov equation (A + BK)X + X(A + BK) T + 4I 2 = 0, we obtain the nonlinear mapping
One can check that g is invertible, and the cost function after applying this mapping becomes
By standard techniques in convex analysis, one can show that h(Y ) is convex. Thanks to the smooth bijection g, minimizing J(K) is now equivalent to minimizing the convex function h(Y ). As a result, any stationary point of J(K) is globally optimal, and in fact, can be shown to be unique. This motivating example demonstrates how one can utilize a proper change of variables to certify the global optimality of stationary points via convex analysis for policy optimization. In the next subsection, we propose a general framework for nonconvex policy optimization that can accommodate a much wider range of benchmark problems admitting convex reformulations.
Consider a policy optimization problem where the objective is f : D → R, with D ⊆ R d being its domain. To study the landscape of f , we resort to its strict and non-strict epigraphs defined by
, the mapping Φ directly outputs γ in the first component).
The notion of ECL gives extensive flexibility by 1) adding an extra variable to lift the epigraph to a higher dimension, 2) relaxing the projection of the lifted set to sit between the strict epigraph and the closure of the non-strict epigraph (chain of inclusion), and 3) introducing an auxiliary set G aux to extend the convex image under a diffeomorphism. The extra variable ξ often corresponds to Lyapunov variables, and G aux is often related to similarity transformations of dynamic policies.
The interested reader may wonder why we need such a peculiar chain of inclusion in (2a). A simpler and more straightforward requirement for the lifting process might be
(3)
Evidently, (2a) includes (3) as a special case. In Section 4, we will present an ECL for outputfeedback H ∞ control where the more general (2a) is necessary, which is largely due to the intricacy between strict and non-strict LMIs in the convex reformulations of control problems. This intricacy is important for global optimality, but has been less emphasized before since classical results often focused on suboptimal controller design; see (Zheng et al., 2023(Zheng et al., , 2024) ) for details. An immediate benefit of ECL is that we can reformulate the minimization of f (x) over x ∈ D as a convex problem.
Theorem 2.1 Let f : D → R be continuous and equipped with an
Thanks to the diffeomorphism Φ, the proof is straightforward but requires careful reasoning about (non-)strict epigraphs; see Zheng et al. (2024, Theorem 3.1) for a detailed proof. In Theorem 2.1, the function f can be nonsmooth and nonconvex, but the existence of an ECL reveals its hidden convexity in the sense that optimizing f (x) over x ∈ D is equivalent to a convex problem. Theorem 2.1 provides the rationale behind convex re-parameterizations of many control problems (Scherer and Weiland, 2015;Boyd et al., 1994). Note that we only guarantee an infimum instead of a minimum in Theorem 2.1 (the infimum may not always be achieved in H ∞ control).
In addition to convex reformulation as shown in Theorem 2.1, the existence of an ECL can further reveal global optimality of certain first-order stationary points for the potentially nonconvex and nonsmooth function f . This allows us to optimize f (x) by direct local search without knowing the particular form of the ECL, which is particularly important for learning-based model-free control.
Before proceeding, we re-emphasize that the chain of inclusion (2a) is critical to the construction of ECL in many control problems. This chain of inclusion allows existence of points (x, f (x)) that are not covered by π x,γ (L lft ), as well as members of π x,γ (L lft ) that are only accumulation points of epi ≥ (f ). We introduce the notion of (non-)degeneracy to characterize the former type of points.
The set of non-degenerate points in D will be denoted by D nd .
In Section 4.1, we will show that all stabilizing state-feedback policies for LQR and H ∞ control are non-degenerate using standard ECL constructions. We will also see that for output-feedback H ∞ control, the set of non-degenerate policies defined in our previous work (Tang and Zheng, 2023) corresponds to the set of non-degenerate points per Definition 3.1 under the ECL framework.
We now present one main technical result of this paper, which provides global optimality certificates for stationary points that are non-degenerate.
The proof of Theorem 3.1 has a strong geometric intuition, but the details are technically involved, which are given in Zheng et al. (2024, Section 3.3). Theorem 3.1 guarantees that nondegenerate stationarity implies global optimality for any subdifferentially regular function with an ECL. Subdifferentially regular functions are a very large class of functions, covering all optimal and robust control problems discussed in Section 4.
We now provide a corollary for the case where (3) holds for the ECL. As mentioned before, many state-feedback and full-order output-feedback controller synthesis problems are nonconvex in their natural forms but admit "convex reformulations" in terms of LMIs using a suitable change of variables. We argue that our notion of ECL presents a unified treatment for many of these convex reformulations. In Section 4, we will present some ECL construction details for benchmark optimal and robust control problems. We point out that, despite the wide use of convex reformulations in control, exact constructions of ECL require special care, especially for output-feedback control problems (due to the subtleties between strict and non-strict LMIs).
Remark 3.1 (Degenerate points and saddles) By (2a), π x,γ (L lft ) may not cover the whole nonstrict epigraph. As a result, Theorem 3.1 does not provide global optimality guarantees for degenerate stationary points x ∈ D\D nd since they cannot be covered by the convex reparameterization. 4 Suboptimal saddle points for f might exist even when equipped with an ECL. Indeed, it has been revealed that LQG policy optimization has strictly sub-optimal saddle points (Tang et al., 2023, Thereom 5) (Zheng et al., 2022, Theorem 2), which are all degenerate per Definition 3.1.
3. Subdifferential regularity allows us to relate the Clarke subdifferential with ordinary directional derivatives. 4. In this sense, some classical LMI reformulations are not "equivalent" to the original control problems, especially for output-feedback cases.
Notations: A K,X := (A + BK)X + P (A + BK) T ; A t K,P := (A + BK) T P + P (A + BK); A X,Y := AX + BY + (AX + BY ) T .
In this section, we present the ECL constructions for several benchmark optimal and robust control problems. Theorems 2.1 and 3.1 can then be directly applied to their policy optimization formulations. Due to space limitations, we omit the mathematical justifications of these ECL constructions, and refer interested readers to Zheng et al. (2023Zheng et al. ( , 2024) ) for detailed proofs.
Consider a continuous-time LTI system
where x(t) ∈ R n is the state variable, u(t) ∈ R m is the control input, and w(t) ∈ R n is the disturbance on the system process. We introduce B w ∈ R n×n for a unified treatment of LQR and H ∞ control in this section. z(t) represents the performance signal, where Q ⪰ 0 and R ≻ 0. We assume (A, B) is controllable and (Q 1/2 , A) is observable.
For both LQR and H ∞ control, their cost values depend on B w only via B w B T w . We thus define W := B w B T w , and assume B w = W 1/2 without loss of generality. We also assume that W ≻ 0. We consider the class of state-feedback policies of the form u(t) = Kx(t), with K ∈ R m×n , to regulate z(t) under the influence of w(t). The set of stabilizing state-feedback policies, parameterized by K, is then K := K ∈ R m×n max i Re λ i (A + BK) < 0 .
In the LQR problem, w(t) is assumed to be a white Gaussian noise with unit intensity. The policy optimization formulation for LQR is then (see, e.g., Mohammadi et al. (2022)),
where X K is the positive semidefinite solution to the Lyapunov equation (A + BK)X K + X K (A + BK) + W = 0. It is known that J LQR (K) is smooth and nonconvex, but has a unique 5 stationary point (which is globally optimal) and is gradient dominated on any sublevel set (Mohammadi et al., 2022). These nice landscape properties are closely related to the hidden convexity of ( 5).
The ECL construction for LQR policy optimization is presented in the second column of Table 1. Specifically, the construction consists of three steps:
Step 1: Lifting. We first define the lifted set L LQR , as delineated in the second row of Table 1. By the theory of linear quadratic control, we have K ∈ K and γ ≥ J LQR (K) if and only if there exists X such that (K, γ, X) ∈ L LQR . This further implies that π K,γ (L LQR ) = epi ≥ (J LQR ).
Step 2: Convex set. We define the convex set F LQR as given in the third row of Table 1, and let the auxiliary set be G LQR = {0}. The first three constraints in the definition of F LQR are obviously convex, and the convexity of the last inequality follows by the joint convexity of tr(QX + X -1 Y T RY ) with respect to (X, Y ).
Step 3: Diffeomorphism. We employ the classical change of variables Y = KX and define Φ LQR (K, γ, X) = (γ, KX, X), where (KX, X) represents the variable ζ 1 in ECL. This mapping naturally satisfies (2b). We do not need ζ 2 here as no similarity transformation exists. One can check by standard calculation that F LQR = Φ LQR (L LQR ), and that Φ LQR admits an inverse on F LQR given by 5). The ECL framework then immediately implies the following well-known results: 1. Theorem 2.1 justifies that the LQR problem ( 5) can be reformulated as a convex program: 6 min
and their optimal solutions K * and (γ * , Y * , X * ) are related by K * = Y * X * -1 and γ * = J LQR (K * ). The policy optimization for LQR can be viewed as a convex problem in disguise. 2. Any stationary point K ⋆ of J LQR is globally optimal, confirmed by π K,γ (L LQR ) = epi ≥ (J LQR )
and Corollary 3.1.
We mention that existing literature has further proved that J LQR (K) has a unique stationary point, is coercive, and is L-smooth and gradient dominated over any sublevel set (Mohammadi et al., 2022;Watanabe and Zheng, 2025). These properties are fundamental to establishing global convergence of direct policy search and their model-free extensions for solving LQR (Malik et al., 2020;Mohammadi et al., 2022). It would be interesting to investigate how to refine our ECL framework to cover these properties.
In state-feedback H ∞ control, we consider w(t) as adversarial disturbance with bounded energy, and the goal is to minimize the maximum energy gain from the disturbance w(t) to the performance signal z(t). It is a standard result in robust control that the state-feedback H ∞ control problem can be formulated as inf
5. This uniqueness is due to W ≻ 0; see Watanabe and Zheng (2025) for details on non-unique optimal LQR gains. 6. The infima for both the original policy optimization and the convex reformulation can be achieved, due to the coerciveness of JLQR(K) and the compactness of (Y, X) (γ, Y, X) ∈ FLQR for any given γ > 0.
where T zw (K, s) is the transfer matrix from w(t) to z(t) when the policy u(t) = Kx(t) is applied, and ∥ • ∥ H∞ denotes the H ∞ norm. Note that the infimum of (6) may not be attained. The H ∞ policy optimization problem ( 6) is nonconvex and nonsmooth, but it admits a convex reformulation (Scherer and Weiland, 2015). It has been recently revealed in Guo and Hu (2022) that for the discrete-time version of (6), any Clarke stationary point is globally optimal. Our aim here is to construct an ECL for (6), which then allows us to draw conclusions from Theorems 2.1 and 3.1 directly. The construction process is very similar to the LQR case, and is delineated in the third column of Table 1. One key difference is that we employ the bounded real lemmas to handle the H ∞ case.
Specifically, our ECL construction consists of the following steps:
Step 1: Lifting. We define the lifted set L ∞ = (K, γ, P ) L ∞ ⪯ 0, P ≻ 0 , where P is an extra Lyapunov variable, and the matrix L ∞ is given in the second row of Table 1.
Step 2: Convex set. We define a convex set
Step 3: Diffeomorphism. We employ the classical change of variables Y = KP -1 , X = P -1 and introduce the mapping Φ ∞ (K, γ, P ) = (γ, KP -1 , P -1 ), where (KP -1 , P -1 ) represents the variable ζ 1 . Similar to the LQR case, no auxiliary variable ζ 2 is needed.
For the construction above, we have the following results.
Proposition 4.1 Consider the state-feedback H ∞ policy optimization problem (6), where (A, B) is controllable, B w has full row rank, and Q ≻ 0, R ≻ 0.
1. For any K ∈ R m×n and γ ∈ R, we have K ∈ K and γ ≥ J ∞ (K) if and only if there exists P such that (K, γ, P ) ∈ L ∞ . This further implies π K,γ (L ∞ ) = epi ≥ (J ∞ ).
2. The mapping Φ ∞ is a C ∞ diffeomorphism between the lifted set L ∞ and the convex set F ∞ .
The proof is not very difficult, but one needs to be careful about some technical subtleties in (non)-strict Riccati inequalities. The details are provided in Zheng et al. (2024, Appendix C.4). Proposition 4.1 guarantees that (L ∞ , F ∞ , {0}, Φ ∞ ) is an ECL of J ∞ (K) in (6). For this ECL, we further have the following nice results, which are consequences of Theorem 2.1 and Corollary 3.1, and the fact that J ∞ (K) is sudifferentially regular.
Theorem 4.1 Under the conditions of Proposition 4.1, the following statements hold. 1. Problem (6) is equivalent to the convex problem inf (γ,Y,X)∈F∞ γ. 2. All stabilizing policies K ∈ K are non-degenerate with respect to the ECL (L ∞ , F ∞ , {0}, Φ ∞ ). 3. Any Clarke stationary point of (6) is globally optimal.
We note that the global optimality of Clarke stationary points for (6) has not been reported before. This is the continuous-time counterpart of the discrete-time result in Guo and Hu (2022). We also note that the infimum of (6) may not be achieved, in which case the Clarke stationary point does not exist. An explicit SISO example is provided in Zheng et al. (2024, Appendix C.2).
Remark 4.1 The diffeomorphisms Φ LQR and Φ ∞ are essentially in the same form and follow from the classical change of variable K = Y X -1 (Khargonekar and Rotea, 1991;Boyd et al., 1994;Bernussou et al., 1989). This change of variable is able to linearize many bilinear matrix inequalities that appear in state-feedback control problems, most of which are related to the Lyapunov inequality (A + BK)X + X(A + BK) T ≺ 0 (see Boyd et al., 1994, Chapter 7 for a historical perspective). As we will see in the next subsection, the linearization for dynamic output-feedback policies turns out to be much more complicated, and we will utilize the techniques in Scherer et al. (1997); Scherer and Weiland (2015) to construct ECLs for output-feedback H ∞ control. □
In this subsection, we show that ECL is also applicable to policy optimization with dynamic outputfeedback policies. Due to space limitations, we only treat the output-feedback H ∞ control problem in this paper; see Zheng et al. (2023Zheng et al. ( , 2024) ) for the case of LQG control. Consider an LTI system with partial observations,
where y(t) ∈ R p is the vector of measured outputs available for feedback control, and w(t) ∈ R n , v(t) ∈ R p are the disturbances on the system process and measurement at time t. We define
and, without loss of generality, assume B w = W 1/2 and D v = V 1/2 . We adopt the same performance signal z(t) in (4). The following assumption is standard.
Assumption 4.1 (A, B) and (A, W 1/2 ) are controllable, and (C, A) and (Q 1/2 , A) are observable. Moreover, Q ⪰ 0, W ⪰ 0 and R ≻ 0, V ≻ 0.
To properly regulate z(t), we consider full-order dynamic output-feedback policies of the form
where ξ(t) ∈ R n is the internal state, and A K , B K , C K and D K are matrices of proper dimensions that specify the policy dynamics. The matrix K is used for parameterizing the dynamic policies. By closing the feedback loop, we can write the transfer matrix from the disturbance d(t) = w T (t) v T (t) T to the performance signal z(t) in the following form:
where A cl (K), B cl (K), C cl (K) and D cl (K) are certain matrix-valued functions that characterize the state-space model of the closed-loop system (see Zheng et al., 2023, Appendix A.5 for details). Now consider a standard output-feedback H ∞ policy optimization problem,
with C n denoting the set of internally stabilizing dynamic policies C n = K A cl (K) is stable . It is known that the policy optimization problem ( 8) is nonsmooth and nonconvex, which has complicated landscape properties. Our ECL construction for J ∞,n (K) is based on the change of variables given in Scherer et al. (1997); Scherer and Weiland (2015), and the details are as follows:
Step 1: Lifting. We first introduce the lifted set L ∞ by
where P 12 denotes the n × n submatrix of P corresponding to the first n rows and last n columns. The extra variable P plays the role of the lifting variable in ECL.
Step 2: Convex and auxiliary sets. We let the convex set be
where (Λ, X, Y ) corresponds to ζ 1 , and M (γ, Λ, X, Y ) is an affine operator whose definition is omitted here due to space limitations. The auxiliary set is GL n = T ∈ R n×n det T ̸ = 0 .
Step 3: Diffeomorphism. We define the mapping Φ ∞ by
, (P -1 ) 11 , P 11 , P 12 , (K, γ, P ) ∈ L ∞ ,
where Φ M = P 12 B K C(P -1 ) 11 + P 11 BC K (P -1 ) 21 + P 11 (A + BD K C)(P -1 ) 11 + P 12 A K (P -1 ) 21 , Φ H = P 11 BD K + P 12 B K , and Φ F = D K C(P -1 ) 11 + C K (P -1 ) 21 .
The following proposition justifies that (L ∞ , F ∞ , GL n , Φ ∞ ) is an ECL for J ∞,n (K). The proof is technically involved and is given in the report Zheng et al. (2024, Appendix D); the main difficulty lies in that (3) does not hold anymore, and we need to establish the chain of inclusion (2a).
Proposition 4.2 Under Assumption 4.1, we have i
Then, by Theorems 2.1 and 3.1, we get the following corollary.
Corollary 4.1 Under Assumption 4.1, the output-feedback H ∞ policy optimization problem (8) is equivalent to a convex problem in the sense that inf K∈Cn J ∞,n (K) = inf (γ,Λ,X,Y )∈F ∞ γ. Furthermore, for a Clarke stationary point K ∈ C n (i.e., 0 ∈ ∂J ∞,n (K)), if K is non-degenerate in the sense of Definition 3.1, then it is globally optimal for (8).
We can now see that our ECL framework covers the output-feedback H ∞ policy optimization problem as a special case, and provides global optimality certificates for non-degenerate Clarke stationary points. Note that the equivalence to the convex reformulation is essentially the same as (Scherer and Weiland, 2015, Chapter 4.2.3), but the global optimality of non-degenerate H ∞ policies cannot be derived from (Scherer and Weiland, 2015, Chapter 4.2.3) due to its use of strict LMIs. Finally, we remark that our ECL can also cover a class of distributed control problems under the condition of quadratic invariance (Furieri et al., 2020a,b); see Zheng et al. (2024, Section 4.4) for details.
This paper introduced the ECL framework to reveal hidden convexity in nonconvex and potentially nonsmooth policy optimization for optimal and robust control. Particularly, we have shown that, with the existence of an ECL for nonconvex policy optimization, all non-degenerate stationary policies are globally optimal. We have built explicit ECLs for LQR, state feedback H ∞ control, and dynamic output-feedback H ∞ control. We hope the ECL framework will be useful for analyzing nonconvex problems in other areas beyond control. We are also interested in developing principled first-order policy optimization algorithms that leverage both convex and nonconvex information within the ECL framework.
© 2025 Y. Zheng, C.-F. Pai & Y. Tang.
We allow d2 = 0, in which case we adopt the convention Gaux = {0} and identify Fcvx × {0} with Fcvx.
Some materials rely on the notion of Clarke subdifferential, which extends subdifferential to nonconvex nonsmooth functions. We refer the readers to Clarke (1990) for details, orZheng et al. (2023, Appendix B) for a brief review.