An official website of the United States government
This paper investigates data-efficient methods for learning robust control policies. Reinforcement learning has emerged as an effective approach to learn control policies by interacting directly with the plant, but it requires a significant number of example trajectories to converge to the optimal policy. Combining model-free reinforcement learning with model-based control methods achieves better data-efficiency via simultaneous system identification and controller synthesis. We study a novel approach that exploits the existence of approximate physics models to accelerate the learning of control policies. The proposed approach consists of iterating through three key steps: evaluating a selected policy on the real-world plant and recording trajectories, building a Gaussian process model to predict the reality-gap of a parametric physics model in the neighborhood of the selected policy, and synthesizing a new policy using reinforcement learning on the refined physics model that most likely approximates the real plant. The approach converges to an optimal policy as well as an approximate physics model. The real world experiments are limited to evaluating only promising candidate policies, and the use of Gaussian processes minimizes the number of required real world trajectories. We demonstrate the effectiveness of our techniques on a set of simulation case-studies using OpenAI gym environments.
more »
« less
Closed-loop control of direct ink writing via reinforcement learning
Enabling additive manufacturing to employ a wide range of novel, functional materials can be a major boost to this technology. However, making such materials printable requires painstaking trial-and-error by an expert operator, as they typically tend to exhibit peculiar rheological or hysteresis properties. Even in the case of successfully finding the process parameters, there is no guarantee of print-to-print consistency due to material differences between batches. These challenges make closed-loop feedback an attractive option where the process parameters are adjusted on-the-fly. There are several challenges for designing an efficient controller: the deposition parameters are complex and highly coupled, artifacts occur after long time horizons, simulating the deposition is computationally costly, and learning on hardware is intractable. In this work, we demonstrate the feasibility of learning a closed-loop control policy for additive manufacturing using reinforcement learning. We show that approximate, but efficient, numerical simulation is sufficient as long as it allows learning the behavioral patterns of deposition that translate to real-world experiences. In combination with reinforcement learning, our model can be used to discover control policies that outperform baseline controllers. Furthermore, the recovered policies have a minimal sim-to-real gap. We showcase this by applying our control policy in-vivo on a single-layer printer using low and high viscosity materials.
more »
« less
- PAR ID:
- 10376677
- Date Published:
- Journal Name:
- ACM Transactions on Graphics
- Volume:
- 41
- Issue:
- 4
- ISSN:
- 0730-0301
- Page Range / eLocation ID:
- 1 to 10
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Aerosol jet printing is a compelling technology for hybrid electronics, combining digital and noncontact patterning with broad materials compatibility, resolution as fine as ≈10 microns, and a high standoff distance of 1–5 mm. Despite its growing popularity in research environments, a robust process understanding and improved manufacturing control are essential for achieving the reliability and predictability required for broader adoption in advanced applications. Herein, recent developments in process monitoring using in‐line light scattering measurements are discussed, including their mechanistic foundations, experimental validation, relevance for process control and reliability, and value as a diagnostic tool for fundamental studies. Experimental measurements confirm the correlation between measured light scattering and deposition rate. Building on this platform, feedback from the real‐time measurement is coupled with printer software to support automated closed‐loop control via a simple proportional‐integral‐derivative software control loop. Combined with the utility of these measurements as a diagnostic to accelerate ink formulation and support fundamental process science experiments, this in‐line measurement provides a useful tool to improve print reliability with the potential to advance the adoption and capabilities of this method in conformal, flexible, and hybrid electronics applications.more » « less
-
K. Ellis, W. Ferrell (Ed.)Fused deposition modeling (FDM) is one of the widely used additive manufacturing (AM) processes but shares major shortcomings typical due to its layer-by-layer fabrication. These challenges (poor surface finishes, presence of pores, inconsistent mechanical properties, etc.) have been attributed to FDM input process parameters, machine parameters, and material properties. Deep learning, a type of machine learning algorithm has proven to help reveal complex and nonlinear input-output relationships without the need for the underlying physics. This research explores the power of multilayer perceptron deep learning algorithm to create a prediction model for critical input process parameters (layer thickness, extrusion temperature, build temperature, build orientation, and print speed) to predict three functional output parameters (dimension accuracy, porosity, and tensile strength) of FDM printed part. A fractional factorial design of experiment was performed and replicated three times per run (n=3). The number of neurons for the hidden layers, learning rate, and epoch were varied. The computational run time, loss function, and root mean square error (RMSE) were used to select the best prediction model for each FDM output parameter. The findings of this work are being extended to online monitoring and real-time control of the AM process enabling an AM digital twin.more » « less
-
K. Ellis, W. Ferrell (Ed.)Fused deposition modeling (FDM) is one of the widely used additive manufacturing (AM) processes but shares major shortcomings typical due to its layer-by-layer fabrication. These challenges (poor surface finishes, presence of pores, inconsistent mechanical properties, etc.) have been attributed to FDM input process parameters, machine parameters, and material properties. Deep learning, a type of machine learning algorithm has proven to help reveal complex and nonlinear input-output relationships without the need for the underlying physics. This research explores the power of multilayer perceptron deep learning algorithm to create a prediction model for critical input process parameters (layer thickness, extrusion temperature, build temperature, build orientation, and print speed) to predict three functional output parameters (dimension accuracy, porosity, and tensile strength) of FDM printed part. A fractional factorial design of experiment was performed and replicated three times per run (n=3). The number of neurons for the hidden layers, learning rate, and epoch were varied. The computational run time, loss function, and root mean square error (RMSE) were used to select the best prediction model for each FDM output parameter. The findings of this work are being extended to online monitoring and real-time control of the AM process enabling an AM digital twin.more » « less
-
With the rapid development of deep reinforcement learning technology, it gradually demonstrates excellent potential and is becoming the most promising solution in the robotics. However, in the smart manufacturing domain, there is still not too much research involved in dynamic adaptive control mechanisms optimizing complex processes. This research advances the integration of Soft Actor-Critic (SAC) with digital twins for industrial robotics applications, providing a framework for enhanced adaptive real-time control for smart additive manufacturing processing. The system architecture combines Unity’s simulation environment with ROS2 for seamless digital twin synchronization, while leveraging transfer learning to efficiently adapt trained models across tasks. We demonstrate our methodology using a Viper X300s robot arm with the proposed hierarchical reward structure to address the common reinforcement learning challenges in two distinct control scenarios. The results show rapid policy convergence and robust task execution in both simulated and physical environments demonstrating the effectiveness of our approach.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1145/3528223.3530144
