More Like this

1. Monitoring of DE-GMAW process in human–robot collaboration

   https://doi.org/10.1007/s40194-025-01936-4  

   Cao, Yue; Chen, Heping; Zhang, YuMing (May 2025, Welding in the World)

   Double-Electrode Gas Metal Arc Welding (DE-GMAW) improves traditional GMAW by adding a non-consumable tungsten electrode, creating a bypass loop that decouples heat input and deposition rate. The bypass arc, critical for establishing the bypass loop, is affected by the bypass electrode position in both horizontal and vertical directions. However, the impact of the bypass electrode positioning has not been studied. This work focuses on monitoring human operations in DE-GMAW within a human-robot collaboration (HRC) setting, aiming to understand the process. Initially, the impact of bypass electrode position on arc morphology and metal transfer was studied, revealing the diversity of the process and the importance of precise electrode positioning. Subsequently, a convolutional neural network was trained using augmented data to accurately detect essential positional information from welding images, thereby determining the optimal operational positioning during human operation. Finally, the relationship between bypass arc voltage and position was quantified using Gaussian Process Regression (GPR), showing that this signal can effectively reflect the process state. This study advances the understanding of DE-GMAW and human operational intelligence, laying a foundational basis for automating the process. 

   more » « less
2. Robotizing double-electrode GMAW process through learning from human welders

   https://doi.org/10.1016/j.jmapro.2023.12.008  

   Yu, Rui; Cao, Yue; Martin, Jennifer; Chiang, Otto; Zhang, YuMing (January 2024, Journal of Manufacturing Processes)

   Gas metal arc welding (GMAW) is the most robotized arc welding process and most widely used process for wire arc additive manufacturing (WAAM). Double-electrode GMAW (DE-GMAW) is its novel modification achieved by adding a second/bypass electrode. It provides the capability to freely adjust the base metal current (heat input) without changing the wire current (mass input). However, it must be robotized in order to be adaptive to manufacturing variations and constraints. Due to the complexity of the process, we propose learning from human welders through their attempts to adjust the bypass electrode in collaboration with the robotized GMAW. To generalize human success using a follower robot/ surrogate, the distance between the wire and bypass electrode is proposed as the process state to quantify human observation and operation. As such, Inertial Measurement Unit (IMU) sensors are integrated to track the wire and bypass electrode operated by both the lead robot and the human welder. To interpret human adjustments per arc observation, a Convolutional Neural Network (CNN) is employed to process arc images and calculate the distance. The automatically obtained distance labels from IMU signals are used to train the CNN; however, they are noisy and inaccurate. Thus, the CNN is finely tuned through transfer learning using manually labeled distances. With accurately automatically calculated distances from the finely tuned CNN for all the data, the demonstrations from human welders are analyzed. The generalized knowledge is implemented by a robotic surrogate, substituting for the human welder to fully automate the DE-GMAW. Experiments demonstrated the superior performance of the fully robotized, and adaptively controlled, DE-GMAW process. 

   more » « less
3. Deep-learning based supervisory monitoring of robotized DE-GMAW process through learning from human welders

   https://doi.org/10.1007/s40194-023-01635-y  

   Yu, Rui; Cao, Yue; Martin, Jennifer; Chiang, Otto; Zhang, YuMing (April 2024, Welding in the World)

   Double-electrode gas metal arc welding (DE-GMAW) modifies GMAW by adding a second electrode to bypass a portion of the current flowing from the wire. This reduces the current to, and the heat input on, the workpiece. Successful bypassing depends on the relative position of the bypass electrode to the continuously varying wire tip. To ensure proper operation, we propose robotizing the system using a follower robot to carry and adaptively adjust the bypass electrode. The primary information for monitoring this process is the arc image, which directly shows desired and undesired modes. However, developing a robust algorithm for processing the complex arc image is time-consuming and challenging. Employing a deep learning approach requires labeling numerous arc images for the corresponding DE-GMAW modes, which is not practically feasible. To introduce alternative labels, we analyze arc phenomena in various DE-GMAW modes and correlate them with distinct arc systems having varying voltages. These voltages serve as automatically derived labels to train the deep-learning network. The results demonstrated reliable process monitoring. 

   more » « less
4. Generative adversarial networks (GAN) model for dynamically adjusted weld pool image toward human-based model predictive control (MPC)

   https://doi.org/10.1016/j.jmapro.2025.02.053  

   Li, Tianpu; Cao, Yue; Ye, Qiang; Zhang, YuMing (May 2025, Journal of Manufacturing Processes)

   Gas Metal Arc Welding (GMAW) is a critical industrial technique known for its high productivity, flexibility, and adaptability to automation. Despite the significant advancements in robotic welding, challenges remain in fully automating the arc welding process, particularly due to the complex dynamics of the weld pool associated with GMAW. A human-robot collaborative (HRC) system where humans operate robots may conveniently provide the needed adaptive control to the complex GMAW. While in conventional HRC systems humans receive process feedback to make adaptive adjustments, we propose provide humans with predictive future feedback to further ease the human decision and reduce the needed skills/trainings. To this end, this study explores the integration of deep learning models, specifically Generative Adversarial Networks (GANs) combined with Gated Recurrent Units (GRUs), to model and predict the dynamic behavior of the weld pool during GMAW. By leveraging time-series data of torch movement and corresponding weld pool images, the proposed GRU-GAN model generates high-fidelity weld pool images, capturing the intricate relationship between speed variations and weld pool morphology. Through extensive experimentation, including the design of an acceptable Encoder-Decoder structure for the GAN, we demonstrate that incorporating both temporal and speed sequence information significantly enhances the model's predictive capabilities. The findings validate the hypothesis that dynamic torch speed adjustments, akin to those performed by skilled human welders, can be effectively modeled to improve the quality of automated welding processes. Future work will be devoted to human-based model predictive control (MPC) in an HRC environment. 

   more » « less
5. Analysis of weld pool region constituents in GMAW for dynamic reconstruction through characteristic enhancement and LSTM U-Net networks

   https://doi.org/10.1016/j.jmapro.2024.07.084  

   Li, Tianpu; Cao, Yue; Zhang, YuMing (October 2024, Journal of Manufacturing Processes)

   This study aims to extract critical scenes/continents in the weld pool region during gas metal arc welding (GMAW). The scenes considered include the wire, arc, and weld pool, while other secondary ones such as oxides are temporarily excluded. They are critical to understanding, analyzing, monitoring and controlling the welding process, in particular the critical correlation how the welding parameter, arc and weld pool are dynamically correlated. Unfortunately, such fundamental correlation has not been studied and lack of effective ways to simultaneously monitor/extract these scenes is responsible. With the development of optoelectronic devices, weld pool regions can be better imaged. However, because of the nature of the scenes in particular the arc which is formed by ionized gas without a clear boundary and highly dynamic, detecting them using computer vision is challenging. Deep learning is an effective method, but model training usually needs a large number of labels. As manually labeling is expensive, we propose an approach to address this challenge that can train a model from a small, inaccurately labeled dataset. This approach is designed, per the characteristics of the scenes and their dynamics All-in-One Network (AOD-Net) was deployed first for defogging, and then the YOLOX network was utilized to identify regions of interest to reduce the impact of molten metal splashes on image sharpness. Subsequently, we developed a timed segmentation network incorporating the Long Short-Term Memory (LSTM) mechanism into U-Net, which can be used to extract more accurate information about the weld pool by combining the temporal and spatial information in the continuous process of welding at a low cost because our scene of interest is in a continuous and dynamic evolutionary process. After defogging and removing the effects of molten metal spatter, we can obtain information on the dynamics of the weld pool and the weld arc at the same time. Experimental results verified that the trained network could extract the critical boundaries accurately under various welding conditions despite the highly dynamic changes and fuzziness of the views. 

   more » « less