An official website of the United States government
Recent advancements in Digital Twin (DT) technology have opened new avenues for smart manufacturing. These systems increasingly depend on adaptive control mechanisms to optimize complex processes and reduce production wastage. This research presents an innovative approach that integrates Soft Actor-Critic (SAC) Reinforcement Learning (RL) algorithm with DT technology with Robot Operating System 2 (ROS2) to enable real-time adaptive control in robotic manufacturing. Our experimental setup consists of a ViperX 300 S robot arm, in which two distinct Tasks: (1) static target reaching and (2) dynamic target following were implemented for simulating adaptive control of manufacturing process. The innovative system architecture combines Unity game engine’s simulation environment with ROS2 for seamless and robust DT synchronization. We implemented a hierarchical reward structure to address common RL challenges, including local minima avoidance, convergence acceleration, and training stability, while leveraging transfer learning to efficiently adapt trained behavior models across tasks. Experimental results demonstrate rapid policy convergence and robust task execution, with performance metrics including cumulative reward, value loss, policy loss, and entropy validating the effectiveness of the approach. To the best of our knowledge, this is the first study to integrate Unity with ROS2-based DT for real-time synchronization and adaptive physical robot control using RL. Unlike prior works limited to offline or low-frequency simulations, our framework achieves stable 20 ms joint-level synchronization, enabling deployment of learned behaviors directly to physical robotic systems through virtual platform. This work advances the integration of RL with realistic DT framework for industrial and manufacturing robotics applications, providing a framework for enhanced adaptive real-time control in smart additive manufacturing (AM) processes.
more »
« less
This content will become publicly available on May 29, 2026
Digital Twin Synchronization: Bridging the Sim-RL Agent to a Real-Time Robotic Additive Manufacturing Control
With the rapid development of deep reinforcement learning technology, it gradually demonstrates excellent potential and is becoming the most promising solution in the robotics. However, in the smart manufacturing domain, there is still not too much research involved in dynamic adaptive control mechanisms optimizing complex processes. This research advances the integration of Soft Actor-Critic (SAC) with digital twins for industrial robotics applications, providing a framework for enhanced adaptive real-time control for smart additive manufacturing processing. The system architecture combines Unity’s simulation environment with ROS2 for seamless digital twin synchronization, while leveraging transfer learning to efficiently adapt trained models across tasks. We demonstrate our methodology using a Viper X300s robot arm with the proposed hierarchical reward structure to address the common reinforcement learning challenges in two distinct control scenarios. The results show rapid policy convergence and robust task execution in both simulated and physical environments demonstrating the effectiveness of our approach.
more »
« less
- Award ID(s):
- 2348013
- PAR ID:
- 10647295
- Publisher / Repository:
- IEEE
- Date Published:
- ISBN:
- 979-8-3315-1321-4
- Page Range / eLocation ID:
- 1 to 4
- Subject(s) / Keyword(s):
- Robot, Reinforcement Learning, Control
- Format(s):
- Medium: X
- Location:
- Worcester, MA, USA
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract The development of smart materials capable of dynamic shape morphing and rapid responsiveness has garnered significant interest for applications in soft robotics, tissue engineering, programmable materials, and adaptive structures. Hydrogels, owing to their intrinsic biocompatibility and flexibility, are promising candidates for such systems. Embedding micro-scale materials within hydrogel networks can further enhance their mechanical and functional properties. In this study, we present a hybrid fabrication platform that integrates surface acoustic wave (SAW)-based acoustofluidics with digital light processing (DLP) photopolymerization to fabricate smart hydrogel composites with programmable shape-memorable behavior. Using the SAW-induced acoustic potential field, silicon carbide (SiC) micro-whiskers are aligned within a custom UV-curable hydrogel ink and subsequently fixed via high-resolution DLP photopolymerization. This dual-control approach enables independent manipulation of micro-whisker orientation and structural geometry. Numerical simulations and Laser Doppler vibrometry-based validation were employed to characterize the acoustic field. To evaluate shape-memory behavior, the fabricated hydrogels were subjected to dehydration and rehydration cycles. The resulting shape transformations, driven by internal stress gradients within the aligned microparticle framework, enabled humidity-responsive actuation. This work establishes a novel strategy for constructing 4D-printed smart hydrogels, offering a versatile platform for the development of next-generation programmable materials and adaptive structures.more » « less
-
Rapid advances in Digital Twin (DT) provide an unprecedented opportunity to derive data-enabled intelligence for smart manufacturing. However, traditional DT is more concerned about real-time data streaming, dashboard visualization, and predictive analytics, but focuses less on multi-agent intelligence. This limitation hampers the development of agentic intelligence for decentralized decision making in complex manufacturing environments. Therefore, this paper presents a Cognitive Digital Twin (CDT) approach for multi-objective production scheduling through decentralized, collaborative multi-agent learning. First, we propose to construct models of heterogeneous agents (e.g., machines, jobs, automated guided vehicles, and automated storage and retrieval systems) that interact with physical and digital twins. Second, multi-objective optimization is embedded in CDT to align production schedules with diverse and often conflicting objectives such as throughput, task transition efficiency, and workload balance. Third, we develop a multi-agent learning approach to enable decentralized decision making in response to unexpected disruptions and dynamic demands. Each agent operates independently and collaboratively with cognitive capabilities, including perception, learning, and reasoning, to optimize the individual agentic objective while contributing to overarching system-wide goals. Finally, the proposed CDT is evaluated and validated with experimental studies in a learning factory environment. Experimental results demonstrate that CDT improves operational performance in terms of task allocation, resource utilization, and system resilience compared to traditional centralized approaches. This initial study of CDT highlights the potential to bring multi-agent cognitive intelligence into next-generation smart manufacturing.more » « less
-
Prototyping and validating hardware–software components, sub-systems and systems within the intelligent transportation system-of-systems framework requires a modular yet flexible and open-access ecosystem. This work presents our attempt to develop such a comprehensive research and education ecosystem, called AutoDRIVE, for synergistically prototyping, simulating and deploying cyber-physical solutions pertaining to autonomous driving as well as smart city management. AutoDRIVE features both software as well as hardware-in-the-loop testing interfaces with openly accessible scaled vehicle and infrastructure components. The ecosystem is compatible with a variety of development frameworks, and supports both single- and multi-agent paradigms through local as well as distributed computing. Most critically, AutoDRIVE is intended to be modularly expandable to explore emergent technologies, and this work highlights various complementary features and capabilities of the proposed ecosystem by demonstrating four such deployment use-cases: (i) autonomous parking using probabilistic robotics approach for mapping, localization, path-planning and control; (ii) behavioral cloning using computer vision and deep imitation learning; (iii) intersection traversal using vehicle-to-vehicle communication and deep reinforcement learning; and (iv) smart city management using vehicle-to-infrastructure communication and internet-of-things.more » « less
-
Discrete manufacturing systems are complex cyber-physical systems (CPS) and their availability, performance, and quality have a big impact on the economy. Smart manufacturing promises to improve these aspects. One key approach that is being pursued in this context is the creation of centralized software-defined control (SDC) architectures and strategies that use diverse sensors and data sources to make manufacturing more adaptive, resilient, and programmable. In this paper, we present SDCWorks-a modeling and simulation framework for SDC. It consists of the semantic structures for creating models, a baseline controller, and an open source implementation of a discrete event simulator for SDCWorks models. We provide the semantics of such a manufacturing system in terms of a discrete transition system which sets up the platform for future research in a new class of problems in formal verification, synthesis, and monitoring. We illustrate the expressive power of SDCWorks by modeling the realistic SMART manufacturing testbed of University of Michigan. We show how our open source SDCWorks simulator can be used to evaluate relevant metrics (throughput, latency, and load) for example manufacturing systems.more » « less
