More Like this

1. MACHINE LEARNING-BASED ROBOTIC OBJECT DETECTION AND GRASPING FOR COLLABORATIVE ASSEMBLY

   Jianjing Zhang, Chuanping Liu (January 2022, Proc. 2022 International Symposium on Flexible Automation (ISFA))

   Hideki Aoyama; Keiich Shirase (Ed.)

   An integral part of information-centric smart manufacturing is the adaptation of industrial robots to complement human workers in a collaborative manner. While advancement in sensing has enabled real-time monitoring of workspace, understanding the semantic information in the workspace, such as parts and tools, remains a challenge for seamless robot integration. The resulting lack of adaptivity to perform in a dynamic workspace have limited robots to tasks with pre-defined actions. In this paper, a machine learning-based robotic object detection and grasping method is developed to improve the adaptivity of robots. Specifically, object detection based on the concept of single-shot detection (SSD) and convolutional neural network (CNN) is investigated to recognize and localize objects in the workspace. Subsequently, the extracted information from object detection, such as the type, position, and orientation of the object, is fed into a multi-layer perceptron (MLP) to generate the desired joint angles of robotic arm for proper object grasping and handover to the human worker. Network training is guided by forward kinematics of the robotic arm in a self-supervised manner to mitigate issues such as singularity in computation. The effectiveness of the developed method is validated on an eDo robotic arm in a human-robot collaborative assembly case study. 

   more » « less
2. Vision-language model-driven scene understanding and robotic object manipulation

   Liu, S; Zhang, J; Gao, RX; Wang, V; Wang, L (August 2024, IEEE Xplore)

   Humans often use natural language instructions to control and interact with robots for task execution. This poses a big challenge to robots that need to not only parse and understand human instructions but also realise semantic understanding of an unknown environment and its constituent elements. To address this challenge, this study presents a vision-language model (VLM)-driven approach to scene understanding of an unknown environment to enable robotic object manipulation. Given language instructions, a pretrained vision-language model built on open-sourced Llama2-chat (7B) as the language model backbone is adopted for image description and scene understanding, which translates visual information into text descriptions of the scene. Next, a zero-shot-based approach to fine-grained visual grounding and object detection is developed to extract and localise objects of interest from the scene task. Upon 3D reconstruction and pose estimate establishment of the object, a code-writing large language model (LLM) is adopted to generate high-level control codes and link language instructions with robot actions for downstream tasks. The performance of the developed approach is experimentally validated through table-top object manipulation by a robot. 

   more » « less
3. NarraGuide: an LLM-based Narrative Mobile Robot for Remote Place Exploration

   https://doi.org/10.1145/3746059.3747697  

   Hu, Yaxin; Sato, Arissa J; Du, Jingxin; Ye, Chenming; Zhu, Anjun; Praveena, Pragathi; Mutlu, Bilge (September 2025, ACM)

   Robotic telepresence enables users to navigate and experience remote environments. However, effective navigation and situational awareness depend on users’ prior knowledge of the environment, limiting the usefulness of these systems for exploring unfamiliar places. We explore how integrating location-aware LLM-based narrative capabilities into a mobile robot can support remote exploration. We developed a prototype system, called NarraGuide, that provides narrative guidance for users to explore and learn about a remote place through a dialogue-based interface. We deployed our prototype in a geology museum, where remote participants (𝑛 = 20) used the robot to tour the museum. Our findings reveal how users perceived the robot’s role, engaged in dialogue in the tour, and expressed preferences for bystander encountering. Our work demonstrates the potential of LLM-enabled robotic capabilities to deliver location-aware narrative guidance and enrich the experience of exploring remote environments. 

   more » « less
4. Automated Constructability Assessment for Robotics in Construction: Case Study of CANVAS

   https://doi.org/10.1061/9780784485262.070  

   Wang, Ziyi; Hu, Yuqing; Leicht, Robert (March 2024, American Society of Civil Engineers)

   Advances in robotics represent a potential shift in the construction industry. Construction planning is planned based on craft work; it is necessary to emphasize external factors such as construction robotics. Improving constructability can enhance design-phase construction opportunities, thereby expanding the potential scope of robot operations. However, robotics are often neglected concerning constructability. Previous studies on constructability concentrated on human-based construction methods; hence, gaps remain in assessing constructability for robotics. To minimize the barriers in robotic construction, this paper presents a method for using a rule-based framework for robotic constructability assessment checks with the help of BIM. Focusing on CANVAS—a drywall finishing robot—this paper applies a BIM-based object-oriented model integrating with ROS to utilize constructability reasoning about robotic operations. A model of rule-checking for robotics in the case study is demonstrated and tested. The availability of design information in the model containing robotics is discussed, showing the need for assessing robotics-related constructability information to support an automated review of robotic constructability assessment. This paper applies a case study to validate use of the framework for robotic constructability assessment in the design phase, leading to an automated constructability assessment of construction robotics. 

   more » « less
5. Contact Optimization for Non-Prehensile Loco-Manipulation via Hierarchical Model Predictive Control

   https://doi.org/10.1109/ICRA48891.2023.10160507  

   Rigo, Alberto; Chen, Yiyu; Gupta, Satyandra K; Nguyen, Quan (May 2023, IEEE)

   Recent studies on quadruped robots have focused on either locomotion or mobile manipulation using a robotic arm. However, legged robots can manipulate large objects using non-prehensile manipulation primitives, such as planar pushing, to drive the object to the desired location. This paper presents a novel hierarchical model predictive control (MPC) for contact optimization of the manipulation task. Using two cascading MPCs, we split the loco-manipulation problem into two parts: the first to optimize both contact force and contact location between the robot and the object, and the second to regulate the desired interaction force through the robot locomotion. Our method is successfully validated in both simulation and hardware experiments. While the baseline locomotion MPC fails to follow the desired trajectory of the object, our proposed approach can effectively control both object's position and orientation with minimal tracking error. This capability also allows us to perform obstacle avoidance for both the robot and the object during the loco-manipulation task. 

   more » « less