More Like this

1. Integrating Large Language Model for Natural Language-Based Instruction toward Robust Human-Robot Collaboration

   Gao, F; Xia, L; Zhang, J; Liu, S; Wang, L; Gao, R (May 2024, Elsevier)

   Human-Robot Collaboration (HRC) aims to create environments where robots can understand workspace dynamics and actively assist humans in operations, with the human intention recognition being fundamental to efficient and safe task fulfillment. Language-based control and communication is a natural and convenient way to convey human intentions. However, traditional language models require instructions to be articulated following a rigid, predefined syntax, which can be unnatural, inefficient, and prone to errors. This paper investigates the reasoning abilities that emerged from the recent advancement of Large Language Models (LLMs) to overcome these limitations, allowing for human instructions to be used to enhance human-robot communication. For this purpose, a generic GPT 3.5 model has been fine-tuned to interpret and translate varied human instructions into essential attributes, such as task relevancy and tools and/or parts required for the task. These attributes are then fused with perceived on-going robot action to generate a sequence of relevant actions. The developed technique is evaluated in a case study where robots initially misinterpreted human actions and picked up wrong tools and parts for assembly. It is shown that the fine-tuned LLM can effectively identify corrective actions across a diverse range of instructional human inputs, thereby enhancing the robustness of human-robot collaborative assembly for smart manufacturing. 

   more » « less
2. Integrating Symbolic Planning and Reinforcement Learning for Following Temporal Logic Specifications

   https://doi.org/10.1109/IJCNN55064.2022.9892304  

   Xu, Duo; Fekri, Faramarz (July 2022, 2022 International Joint Conference on Neural Networks (IJCNN))

   Teaching a deep reinforcement learning (RL) agent to follow instructions in multi-task environments is a challenging problem. We consider that user defines every task by a linear temporal logic (LTL) formula. However, some causal dependencies in complex environments may be unknown to the user in advance. Hence, when human user is specifying instructions, the robot cannot solve the tasks by simply following the given instructions. In this work, we propose a hierarchical reinforcement learning (HRL) framework in which a symbolic transition model is learned to efficiently produce high-level plans that can guide the agent efficiently solve different tasks. Specifically, the symbolic transition model is learned by inductive logic programming (ILP) to capture logic rules of state transitions. By planning over the product of the symbolic transition model and the automaton derived from the LTL formula, the agent can resolve causal dependencies and break a causally complex problem down into a sequence of simpler low-level sub-tasks. We evaluate the proposed framework on three environments in both discrete and continuous domains, showing advantages over previous representative methods. 

   more » « less
3. Conformal Temporal Logic Planning using Large Language Models

   https://doi.org/10.1145/3769111  

   Wang, Jun; Tong, Jiaming; Tan, Kaiyuan; Vorobeychik, Yevgeniy; Kantaros, Yiannis (September 2025, ACM Transactions on Cyber-Physical Systems)

   This paper addresses temporal logic task planning problems for mobile robots. We consider missions that require accomplishing multiple high-level sub-tasks, expressed in natural language (NL), in a temporal and logical order. To formally define the mission, we treat these sub-tasks as atomic predicates in a Linear Temporal Logic (LTL) formula. We refer to this task specification framework as LTL-NL. Our goal is to design plans, defined as sequences of robot actions, accomplishing LTL-NL tasks. This action planning problem cannot be solved directly by existing LTL planners due to the NL nature of atomic predicates. Therefore, we propose HERACLEs, a hierarchical neuro-symbolic planner that relies on a novel integration of (i) existing symbolic planners generating high-level task plans determining the order at which the NL sub-tasks should be accomplished; (ii) pre-trained Large Language Models (LLMs) to design sequences of robot actions for each sub-task in these task plans; and (iii) conformal prediction acting as a formal interface between (i) and (ii) and managing uncertainties due to LLM imperfections. We show, both theoretically and empirically, that HERACLEs can achieve user-defined mission success rates. We demonstrate the efficiency of HERACLEs through comparative numerical experiments against recent LLM-based planners as well as hardware experiments on mobile manipulation tasks. Finally, we present examples demonstrating that our approach enhances user-friendliness compared to conventional symbolic approaches. 

   more » « less
4. Constrained Behavior Tree Generation for Safe LLM-Driven Robot Navigation

   https://doi.org/10.1109/eIT68936.2026.11670368  

   Perando, Jeremy; Pan, Xunyu (May 2026, 2026 IEEE International Conference on Electro Information Technology (eIT))

   Large language models (LLMs) provide a promising interface for mapping natural-language user instructions to robot actions, but unconstrained generation can produce invalid plans and unsafe behaviors when deployed on physical platforms. In this work, we present a three-level, defense-in-depth architecture that uses constrained behavior trees (BTs) as an intermediate, verifiable control mechanism for LLM-driven robot control. At the high-level layer, an LLM produces a task-level BT candidate in a restricted JSON format, limited to an allowed set of node types, actions, and parameter ranges. The candidate is then statically validated using human-defined rules, including schema checks and semantic safety constraints. Invalid candidates are rejected and re-generated up to a retry limit. At the mid-level layer, a BT interpreter executes the validated tree reactively using symbolic world-state updates and outputs motion primitives. A runtime safety monitor detects hazards such as obstacle proximity and execution stalls, overriding the command stream to trigger STOP or a safe fallback behavior. At the low-level layer, the system maps sensor data from the robot’s perception into symbolic state features and converts motion commands from the upper layer into actuator control signals. Experiments in small-scale navigation scenarios demonstrate reliable task execution while enforcing user-specified safety constraints through validation and runtime monitoring. 

   more » « less
5. Hybrid agentic AI and multi-agent systems in smart manufacturing

   https://doi.org/10.1016/j.jmsy.2026.04.002  

   Farahani, Mojtaba A; Khan, Md Irfan; Wuest, Thorsten (June 2026, Journal of Manufacturing Systems)

   The convergence of Agentic Artificial Intelligence (AI) and Multi-Agent Systems (MAS) enables a new paradigm for intelligent decision-making in Smart Manufacturing Systems (SMS). Traditional MAS architectures emphasize distributed coordination and specialized autonomy, while recent advances in agentic AI driven by Large Language Models (LLMs) introduce higher-order reasoning, planning, and tool orchestration capabilities. This paper presents a hybrid agentic AI and multi-agent framework for a Prescriptive Maintenance (RxM) use case, where LLM-based agents provide strategic orchestration and adaptive reasoning, complemented by rule-based and Small Language Models (SLMs) agents performing efficient, domain-specific tasks on the edge. The proposed framework adopts a layered architecture that consists of perception, preprocessing, analytics, and optimization layers, coordinated through an LLM Planner Agent that manages workflow decisions and context retention. Specialized agents autonomously handle schema discovery, intelligent feature analysis, model selection, and prescriptive optimization, while a human-in-the-loop interface ensures transparency and auditability of generated maintenance recommendations. This hybrid approach enables dynamic model adaptation, transparent decision-making, and cost-aware maintenance scheduling based on data-driven insights. An initial proof-of-concept implementation is validated on two industrial manufacturing datasets. The developed framework is modular and extensible, allowing new agents or domain-specific modules to be integrated seamlessly as system capabilities evolve. The results demonstrate the system’s capability to automatically detect schema, adapt preprocessing pipelines, optimize model performance through adaptive intelligence, and generate actionable, prioritized maintenance recommendations. The framework shows promise in achieving improved robustness, scalability, and explainability for RxM in smart manufacturing, bridging the gap between high-level agentic reasoning and low-level autonomous execution. 

   more » « less