skip to main content

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


  • Home
  • Search Results
  • Page 1 of 2

Search for: All records

Creators/Authors contains: "Zhu, He"

Note: When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).
What is a DOI Number?

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

  1. ; ( , 36th International Conference on Computer Aided Verification (CAV))
    We introduce VELM, a reinforcement learning (RL) framework grounded in verification principles for safe exploration in unknown environments. VELM ensures that an RL agent systematically explores its environment, adhering to safety properties throughout the learning process. VELM learns environment models as symbolic formulas and conducts formal reachability analysis over the learned models for safety verification. An online shielding layer is then constructed to confine the RL agent’s exploration solely within a state space verified as safe in the learned model, thereby bolstering the overall safety profile of the RL system. Our experimental results demonstrate the efficacy of VELM across diverse RL environments, highlighting its capacity to significantly reduce safety violations in comparison to existing safe learning techniques, all without compromising the RL agent’s reward performance. 
    more » « less
    Free, publicly-accessible full text available July 24, 2025
  2. ; ; ; ( , Proceedings of the ACM on Programming Languages)

    Deep reinforcement learning (RL) has led to encouraging successes in numerous challenging robotics applications. However, the lack of inductive biases to support logic deduction and generalization in the representation of a deep RL model causes it less effective in exploring complex long-horizon robot-control tasks with sparse reward signals. Existing program synthesis algorithms for RL problems inherit the same limitation, as they either adapt conventional RL algorithms to guide program search or synthesize robot-control programs to imitate an RL model. We propose ReGuS, a reward-guided synthesis paradigm, to unlock the potential of program synthesis to overcome the exploration challenges. We develop a novel hierarchical synthesis algorithm with decomposed search space for loops, on-demand synthesis of conditional statements, and curriculum synthesis for procedure calls, to effectively compress the exploration space for long-horizon, multi-stage, and procedural robot-control tasks that are difficult to address by conventional RL techniques. Experiment results demonstrate that ReGuS significantly outperforms state-of-the-art RL algorithms and standard program synthesis baselines on challenging robot tasks including autonomous driving, locomotion control, and object manipulation.

     
    more » « less
    Free, publicly-accessible full text available June 20, 2025
  3. ; ; ( , Advances in Neural Information Processing Systems (NeurIPS))
    Goal-conditioned reinforcement learning (RL) is a powerful approach for learning general-purpose skills by reaching diverse goals. However, it has limitations when it comes to task-conditioned policies, where goals are specified by temporally extended instructions written in the Linear Temporal Logic (LTL) formal language. Existing approaches for finding LTL-satisfying policies rely on sampling a large set of LTL instructions during training to adapt to unseen tasks at inference time. However, these approaches do not guarantee generalization to out-of-distribution LTL objectives, which may have increased complexity. In this paper, we propose a novel approach to address this challenge. We show that simple goal-conditioned RL agents can be instructed to follow arbitrary LTL specifications without additional training over the LTL task space. Unlike existing approaches that focus on LTL specifications expressible as regular expressions, our technique is unrestricted and generalizes to ω-regular expressions. Experiment results demonstrate the effectiveness of our approach in adapting goal-conditioned RL agents to satisfy complex temporal logic task specifications zero-shot. 
    more » « less
    Free, publicly-accessible full text available December 10, 2024
  4. ; ; ; ; ( , Carbon Balance and Management)
    Abstract Background

    Timber harvesting and industrial wood processing laterally transfer the carbon stored in forest sectors to wood products creating a wood products carbon pool. The carbon stored in wood products is allocated to end-use wood products (e.g., paper, furniture), landfill, and charcoal. Wood products can store substantial amounts of carbon and contribute to the mitigation of greenhouse effects. Therefore, accurate accounts for the size of wood products carbon pools for different regions are essential to estimating the land-atmosphere carbon exchange by using the bottom-up approach of carbon stock change.

     Results

    To quantify the carbon stored in wood products, we developed a state-of-the-art estimator (Wood Products Carbon Storage Estimator, WPsCS Estimator) that includes the wood products disposal, recycling, and waste wood decomposition processes. The wood products carbon pool in this estimator has three subpools: (1) end-use wood products, (2) landfill, and (3) charcoal carbon. In addition, it has a user-friendly interface, which can be used to easily parameterize and calibrate an estimation. To evaluate its performance, we applied this estimator to account for the carbon stored in wood products made from the timber harvested in Maine, USA, and the carbon storage of wood products consumed in the United States.

     Conclusion

    The WPsCS Estimator can efficiently and easily quantify the carbon stored in harvested wood products for a given region over a specific period, which was demonstrated with two illustrative examples. In addition, WPsCS Estimator has a user-friendly interface, and all parameters can be easily modified.

     
    more » « less
    Full Text Available 
  5. ; ( , ools and Algorithms for the Construction and Analysis of Systems. TACAS 2023. LNCS)
    We present a verification-based learning framework VEL that synthesizes safe programmatic controllers for environments with continuous state and action spaces. The key idea is the integration of program reasoning techniques into controller training loops. VEL performs abstraction-based program verification to reason about a programmatic controller and its environment as a closed-loop system. Based on a novel verification-guided synthesis loop for training, VEL minimizes the amount of safety violation in the proof space of the system, which approximates the worst-case safety loss, using gradient-descent style optimization. Experimental results demonstrate the substantial benefits of leveraging verification feedback for synthesizing provably correct controllers. 
    more » « less
    Full Text Available 
  6. ; ; ; ( , CIKM '22: Proceedings of the 31st ACM International Conference on Information & Knowledge Management)
    Full Text Available 
  7. ; ( , The Tenth International Conference on Learning Representations)
    Deep reinforcement learning (RL) has led to encouraging successes in many challenging control tasks. However, a deep RL model lacks interpretability due to the difficulty of identifying how the model's control logic relates to its network structure. Programmatic policies structured in more interpretable representations emerge as a promising solution. Yet two shortcomings remain: First, synthesizing programmatic policies requires optimizing over the discrete and non-differentiable search space of program architectures. Previous works are suboptimal because they only enumerate program architectures greedily guided by a pretrained RL oracle. Second, these works do not exploit compositionality, an important programming concept, to reuse and compose primitive functions to form a complex function for new tasks. Our first contribution is a programmatically interpretable RL framework that conducts program architecture search on top of a continuous relaxation of the architecture space defined by programming language grammar rules. Our algorithm allows policy architectures to be learned with policy parameters via bilevel optimization using efficient policy-gradient methods, and thus does not require a pretrained oracle. Our second contribution is improving programmatic policies to support compositionality by integrating primitive functions learned to grasp task-agnostic skills as a composite program to solve novel RL problems. Experiment results demonstrate that our algorithm excels in discovering optimal programmatic policies that are highly interpretable. 
    more » « less
    Accepted Manuscript Full Text Available
  8. ; ( , Advances in Neural Information Processing Systems 34 (NeurIPS 2021))
    Differentiable programs have recently attracted much interest due to their interpretability, compositionality, and their efficiency to leverage differentiable training. However, synthesizing differentiable programs requires optimizing over a combinatorial, rapidly exploded space of program architectures. Despite the development of effective pruning heuristics, previous works essentially enumerate the discrete search space of program architectures, which is inefficient. We propose to encode program architecture search as learning the probability distribution over all possible program derivations induced by a context-free grammar. This allows the search algorithm to efficiently prune away unlikely program derivations to synthesize optimal program architectures. To this end, an efficient gradient-descent based method is developed to conduct program architecture search in a continuous relaxation of the discrete space of grammar rules. Experiment results on four sequence classification tasks demonstrate that our program synthesizer excels in discovering program architectures that lead to differentiable programs with higher F1 scores, while being more efficient than state-of-the-art program synthesis methods. 
    more » « less
    Accepted Manuscript Full Text Available
  9. ; ; ; ; ; ( , WSDM '22: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining)
    Full Text Available 
  10. ; ; ; ; ( , Construction and Building Materials)
    Full Text Available