More Like this

1. Early Prediction of Human Intention for Human–Robot Collaboration Using Transformer Network

   https://doi.org/10.1115/1.4064258  

   Zhang, Xinyao; Tian, Sibo; Liang, Xiao; Zheng, Minghui; Behdad, Sara (May 2024, Journal of Computing and Information Science in Engineering)

   Abstract Human intention prediction plays a critical role in human–robot collaboration, as it helps robots improve efficiency and safety by accurately anticipating human intentions and proactively assisting with tasks. While current applications often focus on predicting intent once human action is completed, recognizing human intent in advance has received less attention. This study aims to equip robots with the capability to forecast human intent before completing an action, i.e., early intent prediction. To achieve this objective, we first extract features from human motion trajectories by analyzing changes in human joint distances. These features are then utilized in a Hidden Markov Model (HMM) to determine the state transition times from uncertain intent to certain intent. Second, we propose two models including a Transformer and a Bi-LSTM for classifying motion intentions. Then, we design a human–robot collaboration experiment in which the operator reaches multiple targets while the robot moves continuously following a predetermined path. The data collected through the experiment were divided into two groups: full-length data and partial data before state transitions detected by the HMM. Finally, the effectiveness of the suggested framework for predicting intentions is assessed using two different datasets, particularly in a scenario when motion trajectories are similar but underlying intentions vary. The results indicate that using partial data prior to the motion completion yields better accuracy compared to using full-length data. Specifically, the transformer model exhibits a 2% improvement in accuracy, while the Bi-LSTM model demonstrates a 6% increase in accuracy. 

   more » « less
2. Multi-Task Learning for Intention and Trajectory Prediction in Human–Robot Collaborative Disassembly Tasks

   https://doi.org/10.1115/1.4067157  

   Zhang, Xinyao; Tian, Sibo; Liang, Xiao; Zheng, Minghui; Behdad, Sara (May 2025, Journal of Computing and Information Science in Engineering)

   Abstract Human–robot collaboration (HRC) has become an integral element of many manufacturing and service industries. A fundamental requirement for safe HRC is understanding and predicting human trajectories and intentions, especially when humans and robots operate nearby. Although existing research emphasizes predicting human motions or intentions, a key challenge is predicting both human trajectories and intentions simultaneously. This paper addresses this gap by developing a multi-task learning framework consisting of a bi-long short-term memory-based encoder–decoder architecture that obtains the motion data from both human and robot trajectories as inputs and performs two main tasks simultaneously: human trajectory prediction and human intention prediction. The first task predicts human trajectories by reconstructing the motion sequences, while the second task tests two main approaches for intention prediction: supervised learning, specifically a support vector machine, to predict human intention based on the latent representation, and, an unsupervised learning method, the hidden Markov model, that decodes the latent features for human intention prediction. Four encoder designs are evaluated for feature extraction, including interaction-attention, interaction-pooling, interaction-seq2seq, and seq2seq. The framework is validated through a case study of a desktop disassembly task with robots operating at different speeds. The results include evaluating different encoder designs, analyzing the impact of incorporating robot motion into the encoder, and detailed visualizations. The findings show that the proposed framework can accurately predict human trajectories and intentions. 

   more » « less
3. Mixed-Reality (MR) Enhanced Human–Robot Collaboration: Communicating Robot Intentions to Humans

   https://doi.org/10.3390/robotics14100133  

   Zhang, Kaiyuan; Yan, Yuchen; Jia, Yunyi (October 2025, Robotics)

   Advancements in collaborative robotics have significantly enhanced the potential for human–robot collaboration in manufacturing. To achieve efficient and user-friendly collaboration, prior research has predominantly focused on the robot’s perspective, including aspects such as planning, control, and adaptation. A key approach in this domain has been the recognition of human intentions to inform robot actions. However, true collaboration necessitates bidirectional communication, where both human and robot are aware of each other’s intentions. A lack of transparency in robot actions can lead to discomfort, reduced safety, and inefficiencies in the collaborative process. This study investigates the communication of robot intentions to human operators through mixed reality (MR) and evaluates its impact on human–robot collaboration. A laboratory-based physical human–robot assembly framework is developed, integrating multiple MR-based intention communication strategies. Experimental evaluations are conducted to assess the effectiveness of these strategies. The results demonstrate that conveying robot intentions via MR enhances work efficiency, trust, and user comfort in human–robot collaborative manufacturing. Furthermore, a comparative analysis of different MR-based communication designs provides insights into the optimal approaches for improving collaboration quality. 

   more » « less
4. Multi-Task Learning for Intention and Trajectory Prediction in Human-Robot Collaborative Disassembly Tasks

   https://doi.org/10.1115/DETC2024-143753  

   Zhang, Xinyao; Tian, Sibo; Liang, Xiao; Zheng, Minghui; Behdad, Sara (August 2024, American Society of Mechanical Engineers)

   Abstract Human-robot collaboration (HRC) has become an integral element of many industries, including manufacturing. A fundamental requirement for safe HRC is to understand and predict human intentions and trajectories, especially when humans and robots operate in close proximity. However, predicting both human intention and trajectory components simultaneously remains a research gap. In this paper, we have developed a multi-task learning (MTL) framework designed for HRC, which processes motion data from both human and robot trajectories. The first task predicts human trajectories, focusing on reconstructing the motion sequences. The second task employs supervised learning, specifically a Support Vector Machine (SVM), to predict human intention based on the latent representation. In addition, an unsupervised learning method, Hidden Markov Model (HMM), is utilized for human intention prediction that offers a different approach to decoding the latent features. The proposed framework uses MTL to understand human behavior in complex manufacturing environments. The novelty of the work includes the use of a latent representation to capture temporal dynamics in human motion sequences and a comparative analysis of various encoder architectures. We validate our framework through a case study focused on a HRC disassembly desktop task. The findings confirm the system’s capability to accurately predict both human intentions and trajectories. 

   more » « less
5. Multi-Task Learning for Intention and Trajectory Prediction in Human-Robot Collaborative Disassembly Tasks

   Zhang, Xinyao; Tian, Sibo; Liang, Xiao; Zheng, Minghui; Behdad, Sara (August 2024, Proceedings of the ASME 2024 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC/CIE2024, August 25–28, 2024, Washington, DC, USA.)

   Human-robot collaboration (HRC) has become an integral element of many industries, including manufacturing. A fundamental requirement for safe HRC is to understand and predict human intentions and trajectories, especially when humans and robots operate in close proximity. However, predicting both human intention and trajectory components simultaneously remains a research gap. In this paper, we have developed a multi-task learning (MTL) framework designed for HRC, which processes motion data from both human and robot trajectories. The first task predicts human trajectories, focusing on reconstructing the motion sequences. The second task employs supervised learning, specifically a Support Vector Machine (SVM), to predict human intention based on the latent representation. In addition, an unsupervised learning method, Hidden Markov Model (HMM), is utilized for human intention prediction that offers a different approach to decoding the latent features. The proposed framework uses MTL to understand human behavior in complex manufacturing environments. The novelty of the work includes the use of a latent representation to capture temporal dynamics in human motion sequences and a comparative analysis of various encoder architectures. We validate our framework through a case study focused on a HRC disassembly desktop task. The findings confirm the system's capability to accurately predict both human intentions and trajectories. 

   more » « less