An official website of the United States government
We present a two-stage approach for learning stability-certified neural controllers that achieves a reduction of up to ∼95% in training time compared to the state-of-the-art baseline, which introduced monotonic neural Lyapunov architectures. Our method combines monotonic neural Lyapunov functions with fulfillment priority logic (FPL) to efficiently initialize controllers before formal verification. Traditional approaches for jointly learning controllers and neural Lyapunov functions require computationally expensive mixed-integer linear programming (MILP) or satisfiability modulo theory (SMT) solvers at each training iteration, often taking several hours to converge. We address this bottleneck by leveraging FPL to perform early joint initialization of the controller and Lyapunov networks. Building on the monotonic neural network architecture from the baseline, which guarantees non-negativity and a unique global minimum by construction, our method focuses on efficiently satisfying the remaining property of decreasing along trajectories. Existing works focus on maximizing the region of attraction/convergence of the learned controller. In contrast, leveraging FPL allows us to (1) increase learning efficiency substantially and (2) focus on complementary performance metrics, such as convergence rate and control effort minimization, thereby adding significant specification flexibility. In this paper, we encode an approximate Lyapunov-decrease condition in FPL to pre-train the controller and Lyapunov networks, then apply a MILP-based verification/refinement step. This decouples efficient learning from certificate enforcement and allows the FPL specification to include auxiliary objectives (e.g., convergence rate and control effort), whose influence persists through the final MILP pass. The resulting controllers converge rapidly while admitting formal Lyapunov certificates on standard nonlinear control benchmarks.
more »
« less
Closing the Intent-to-Reality Gap via Fulfillment Priority Logic
Practitioners designing reinforcement learning policies face a fundamental challenge: translating intended behavioral objectives into representative reward functions. This challenge stems from behavioral intent requiring simultaneous achievement of multiple competing objectives, typically addressed through labor-intensive linear reward composition that yields brittle results. Consider the ubiquitous robotics scenario where performance maximization directly conflicts with energy conservation. Such competitive dynamics are resistant to simple linear reward combinations. In this paper, we present the concept of objective fulfillment upon which we build Fulfillment Priority Logic (FPL). FPL allows practitioners to define logical formulae representing their intentions and priorities within multi-objective reinforcement learning. Our novel Balanced Policy Gradient algorithm leverages FPL specifications to achieve up to 500% better sample efficiency compared to Soft Actor Critic. Notably, this work constitutes the first implementation of a non-linear utility scalarization design, intended explicitly for continuous control problems.
more »
« less
- Award ID(s):
- 2238476
- PAR ID:
- 10684882
- Publisher / Repository:
- In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2025)
- Date Published:
- Format(s):
- Medium: X
- Location:
- Hangzhou, China
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
We present a two-stage approach for learning stability-certified neural controllers that achieves a reduction of up to ~95% in training time compared to the state-of-the-art baseline, which introduced monotonic neural Lyapunov architectures. Our method combines monotonic neural Lyapunov functions with fulfillment priority logic (FPL) to efficiently initialize controllers before formal verification. Traditional approaches for jointly learning controllers and neural Lyapunov functions require computationally expensive mixed-integer linear programming (MILP) or satisfiability modulo theory (SMT) solvers at each training iteration, often taking several hours to converge. We address this bottleneck by leveraging FPL to perform early joint initialization of the controller and Lyapunov networks. Building on the monotonic neural network architecture from the baseline, which guarantees non-negativity and a unique global minimum by construction, our method focuses on efficiently satisfying the remaining property of decreasing along trajectories. Existing works focus on maximizing the region of attraction/convergence of the learned controller. In contrast, leveraging FPL allows us to (1) increase learning efficiency substantially and (2) focus on complementary performance metrics, such as convergence rate and control effort minimization, thereby adding significant specification flexibility. In this paper, we encode an approximate Lyapunov-decrease condition in FPL to pre-train the controller and Lyapunov networks, then apply a MILP-based verification/refinement step. This decouples efficient learning from certificate enforcement and allows the FPL specification to include auxiliary objectives (e.g., convergence rate and control effort), whose influence persists through the final MILP pass. The resulting controllers converge rapidly while admitting formal Lyapunov certificates on standard nonlinear control benchmarks.more » « less
-
In traditional reinforcement learning (RL), the learner aims to solve a single objective optimization problem: find the policy that maximizes expected reward. However, in many real-world settings, it is important to optimize over multiple objectives simultaneously. For example, when we are interested in fairness, states might have feature annotations corresponding to multiple (intersecting) demographic groups to whom reward accrues, and our goal might be to maximize the reward of the group receiving the minimal reward. In this work, we consider a multi-objective optimization problem in which each objective is defined by a state-based reweighting of a single scalar reward function. This generalizes the problem of maximizing the reward of the minimum reward group. We provide oracle-efficient algorithms to solve these multi-objective RL problems even when the number of objectives is exponentially large-for tabular MDPs, as well as for large MDPs when the group functions have additional structure. Finally, we experimentally validate our theoretical results and demonstrate applications on a preferential attachment graph MDP.more » « less
-
The expanding role of reinforcement learning (RL) in safety-critical system design has promoted ω-automata as a way to express learning requirements—often non-Markovian—with greater ease of expression and interpretation than scalar reward signals. However, real-world sequential decision making situations often involve multiple, potentially conflicting, objectives. Two dominant approaches to express relative preferences over multiple objectives are: (1)weighted preference, where the decision maker provides scalar weights for various objectives, and (2)lexicographic preference, where the decision maker provides an order over the objectives such that any amount of satisfaction of a higher-ordered objective is preferable to any amount of a lower-ordered one. In this article, we study and develop RL algorithms to compute optimal strategies in Markov decision processes against multiple ω-regular objectives under weighted and lexicographic preferences. We provide a translation from multiple ω-regular objectives to a scalar reward signal that is bothfaithful(maximising reward means maximising probability of achieving the objectives under the corresponding preference) andeffective(RL quickly converges to optimal strategies). We have implemented the translations in a formal reinforcement learning tool,Mungojerrie, and we present an experimental evaluation of our technique on benchmark learning problems.more » « less
-
null (Ed.)Omega-regular properties—specified using linear time temporal logic or various forms of omega-automata—find increasing use in specifying the objectives of reinforcement learning (RL). The key problem that arises is that of faithful and effective translation of the objective into a scalar reward for model-free RL. A recent approach exploits Büchi automata with restricted nondeterminism to reduce the search for an optimal policy for an Open image in new window-regular property to that for a simple reachability objective. A possible drawback of this translation is that reachability rewards are sparse, being reaped only at the end of each episode. Another approach reduces the search for an optimal policy to an optimization problem with two interdependent discount parameters. While this approach provides denser rewards than the reduction to reachability, it is not easily mapped to off-the-shelf RL algorithms. We propose a reward scheme that reduces the search for an optimal policy to an optimization problem with a single discount parameter that produces dense rewards and is compatible with off-the-shelf RL algorithms. Finally, we report an experimental comparison of these and other reward schemes for model-free RL with omega-regular objectives.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Conference Paper:
- The DOI is not currently available.
