More Like this

1. MACHINE LEARNING-BASED ROBOTIC OBJECT DETECTION AND GRASPING FOR COLLABORATIVE ASSEMBLY

   Jianjing Zhang, Chuanping Liu ( January 2022 , Proc. 2022 International Symposium on Flexible Automation (ISFA))

   Hideki Aoyama ; Keiich Shirase (Ed.)

   An integral part of information-centric smart manufacturing is the adaptation of industrial robots to complement human workers in a collaborative manner. While advancement in sensing has enabled real-time monitoring of workspace, understanding the semantic information in the workspace, such as parts and tools, remains a challenge for seamless robot integration. The resulting lack of adaptivity to perform in a dynamic workspace have limited robots to tasks with pre-defined actions. In this paper, a machine learning-based robotic object detection and grasping method is developed to improve the adaptivity of robots. Specifically, object detection based on the concept of single-shot detection (SSD) and convolutional neural network (CNN) is investigated to recognize and localize objects in the workspace. Subsequently, the extracted information from object detection, such as the type, position, and orientation of the object, is fed into a multi-layer perceptron (MLP) to generate the desired joint angles of robotic arm for proper object grasping and handover to the human worker. Network training is guided by forward kinematics of the robotic arm in a self-supervised manner to mitigate issues such as singularity in computation. The effectiveness of the developed method is validated on an eDo robotic arm in a human-robot collaborative assembly case study. 

   more » « less
2. Vision-language model-driven scene understanding and robotic object manipulation

   Liu, S ; Zhang, J ; Gao, RX ; Wang, V ; Wang, L ( August 2024 , IEEE Xplore)

   Humans often use natural language instructions to control and interact with robots for task execution. This poses a big challenge to robots that need to not only parse and understand human instructions but also realise semantic understanding of an unknown environment and its constituent elements. To address this challenge, this study presents a vision-language model (VLM)-driven approach to scene understanding of an unknown environment to enable robotic object manipulation. Given language instructions, a pretrained vision-language model built on open-sourced Llama2-chat (7B) as the language model backbone is adopted for image description and scene understanding, which translates visual information into text descriptions of the scene. Next, a zero-shot-based approach to fine-grained visual grounding and object detection is developed to extract and localise objects of interest from the scene task. Upon 3D reconstruction and pose estimate establishment of the object, a code-writing large language model (LLM) is adopted to generate high-level control codes and link language instructions with robot actions for downstream tasks. The performance of the developed approach is experimentally validated through table-top object manipulation by a robot. 

   more » « less
3. Automated Constructability Assessment for Robotics in Construction: Case Study of CANVAS

   https://doi.org/10.1061/9780784485262.070  

   Wang, Ziyi ; Hu, Yuqing ; Leicht, Robert ( March 2024 , American Society of Civil Engineers)

   Advances in robotics represent a potential shift in the construction industry. Construction planning is planned based on craft work; it is necessary to emphasize external factors such as construction robotics. Improving constructability can enhance design-phase construction opportunities, thereby expanding the potential scope of robot operations. However, robotics are often neglected concerning constructability. Previous studies on constructability concentrated on human-based construction methods; hence, gaps remain in assessing constructability for robotics. To minimize the barriers in robotic construction, this paper presents a method for using a rule-based framework for robotic constructability assessment checks with the help of BIM. Focusing on CANVAS—a drywall finishing robot—this paper applies a BIM-based object-oriented model integrating with ROS to utilize constructability reasoning about robotic operations. A model of rule-checking for robotics in the case study is demonstrated and tested. The availability of design information in the model containing robotics is discussed, showing the need for assessing robotics-related constructability information to support an automated review of robotic constructability assessment. This paper applies a case study to validate use of the framework for robotic constructability assessment in the design phase, leading to an automated constructability assessment of construction robotics. 

   more » « less
4. Contact Optimization for Non-Prehensile Loco-Manipulation via Hierarchical Model Predictive Control

   https://doi.org/10.1109/ICRA48891.2023.10160507  

   Rigo, Alberto ; Chen, Yiyu ; Gupta, Satyandra K ; Nguyen, Quan ( May 2023 , IEEE)

   Recent studies on quadruped robots have focused on either locomotion or mobile manipulation using a robotic arm. However, legged robots can manipulate large objects using non-prehensile manipulation primitives, such as planar pushing, to drive the object to the desired location. This paper presents a novel hierarchical model predictive control (MPC) for contact optimization of the manipulation task. Using two cascading MPCs, we split the loco-manipulation problem into two parts: the first to optimize both contact force and contact location between the robot and the object, and the second to regulate the desired interaction force through the robot locomotion. Our method is successfully validated in both simulation and hardware experiments. While the baseline locomotion MPC fails to follow the desired trajectory of the object, our proposed approach can effectively control both object's position and orientation with minimal tracking error. This capability also allows us to perform obstacle avoidance for both the robot and the object during the loco-manipulation task. 

   more » « less
5. Integrating Large Language Model for Natural Language-Based Instruction toward Robust Human-Robot Collaboration

   Gao, F ; Xia, L ; Zhang, J ; Liu, S ; Wang, L ; Gao, R ( May 2024 , Elsevier)

   Human-Robot Collaboration (HRC) aims to create environments where robots can understand workspace dynamics and actively assist humans in operations, with the human intention recognition being fundamental to efficient and safe task fulfillment. Language-based control and communication is a natural and convenient way to convey human intentions. However, traditional language models require instructions to be articulated following a rigid, predefined syntax, which can be unnatural, inefficient, and prone to errors. This paper investigates the reasoning abilities that emerged from the recent advancement of Large Language Models (LLMs) to overcome these limitations, allowing for human instructions to be used to enhance human-robot communication. For this purpose, a generic GPT 3.5 model has been fine-tuned to interpret and translate varied human instructions into essential attributes, such as task relevancy and tools and/or parts required for the task. These attributes are then fused with perceived on-going robot action to generate a sequence of relevant actions. The developed technique is evaluated in a case study where robots initially misinterpreted human actions and picked up wrong tools and parts for assembly. It is shown that the fine-tuned LLM can effectively identify corrective actions across a diverse range of instructional human inputs, thereby enhancing the robustness of human-robot collaborative assembly for smart manufacturing. 

   more » « less