More Like this

1. Keypoints-Based Adaptive Visual Servoing for Control of Robotic Manipulators in Configuration Space

   https://doi.org/10.1109/IROS55552.2023.10342503  

   Chatterjee, Sreejani; Karade, Abhay C.; Gandhi, Abhinav; Calli, Berk (October 2023, Proceedings of 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS))

   This paper presents a visual servoing method for controlling a robot in the configuration space by purely using its natural features. We first created a data collection pipeline that uses camera intrinsics, extrinsics, and forward kinematics to generate 2D projections of a robot's joint locations (keypoints) in image space. Using this pipeline, we are able to collect large sets of real-robot data, which we use to train realtime keypoint detectors. The inferred keypoints from the trained model are used as control features in an adaptive visual servoing scheme that estimates, in runtime, the Jacobian relating the changes of the keypoints and joint velocities. We compared the 2D configuration control performance of this method to the skeleton-based visual servoing method (the only other algorithm for purely vision-based configuration space visual servoing), and demonstrated that the keypoints provide more robust and less noisy features, which result in better transient response. We also demonstrate the first vision-based 3D configuration space control results in the literature, and discuss its limitations. Our data collection pipeline is available at https://github.com/JaniC-WPI/KPDataGenerator.git which can be utilized to collect image datasets and train realtime keypoint detectors for various robots and environments. 

   more » « less
2. AFT: Appearance-based feature tracking for markerless and training-free shape reconstruction of soft robots

   https://doi.org/10.1109/LRA.2025.3640973  

   Yuan, Shangyuan; Fairchild, Preston; Mei, Yu; Zhou, Xinyu; Tan, Xiaobo (February 2026, IEEE Robotics and Automation Letters)

   Accurate shape reconstruction is essential for precise control and reliable operation of soft robots. Compared to sensor-based approaches, vision-based methods offer advantages in cost, simplicity, and ease of deployment. However, existing vision-based methods often rely on complex camera setups, specific backgrounds, or large-scale training datasets, limiting their practicality in real-world scenarios. In this work, we propose a vision-based, markerless, and training-free framework for soft robotic shape reconstruction that directly leverages the natural surface appearance of the robot. These surface features act as implicit visual markers, enabling a hierarchical matching strategy that decouples local partition alignment from global kinematic optimization. Requiring only an initial 3D reconstruction and kinematic alignment, our method achieves real-time shape tracking across diverse environments while maintaining robustness to occlusions and variations in camera viewpoints. Experimental validation on a continuum soft robot demonstrates an average tip error of 2.6% during real-time operation, as well as stable performance in practical closed-loop control tasks. These results highlight the potential of the proposed approach for reliable, low-cost deployment in dynamic real-world settings. 

   more » « less
3. Eagle: End-to-end Deep Reinforcement Learning based Autonomous Control of PTZ Cameras

   https://doi.org/10.1145/3576842.3582366  

   Sandha, Sandeep Singh; Balaji, Bharathan; Garcia, Luis; Srivastava, Mani (May 2023, IoTDI '23: Proceedings of the 8th ACM/IEEE Conference on Internet of Things Design and Implementation)

   Existing approaches for autonomous control of pan-tilt-zoom (PTZ) cameras use multiple stages where object detection and localization are performed separately from the control of the PTZ mechanisms. These approaches require manual labels and suffer from performance bottlenecks due to error propagation across the multi-stage flow of information. The large size of object detection neural networks also makes prior solutions infeasible for real-time deployment in resource-constrained devices. We present an end-to-end deep reinforcement learning (RL) solution called Eagle1 to train a neural network policy that directly takes images as input to control the PTZ camera. Training reinforcement learning is cumbersome in the real world due to labeling effort, runtime environment stochasticity, and fragile experimental setups. We introduce a photo-realistic simulation framework for training and evaluation of PTZ camera control policies. Eagle achieves superior camera control performance by maintaining the object of interest close to the center of captured images at high resolution and has up to 17% more tracking duration than the state-of-the-art. Eagle policies are lightweight (90x fewer parameters than Yolo5s) and can run on embedded camera platforms such as Raspberry PI (33 FPS) and Jetson Nano (38 FPS), facilitating real-time PTZ tracking for resource-constrained environments. With domain randomization, Eagle policies trained in our simulator can be transferred directly to real-world scenarios2. 

   more » « less
4. Keypoint abstraction using large models for object-relative imitation learning.

   Fang, Xiaolin; Huang, Bo-Ruei; Mao, Jiayuan; Shone, Jasmine; Tenenbaum, Joshua; Lozano-Perez, Tomas; Kaelbling, Leslie (June 2025, Proceedings IEEE International Conference on Robotics and Automation)

   Generalization to novel object configurations and instances across diverse tasks and environments is a critical challenge in robotics. Keypoint-based representations have been proven effective as a succinct representation for capturing essential object features, and for establishing a reference frame in action prediction, enabling data-efficient learning of robot skills. However, their manual design nature and reliance on additional human labels limit their scalability. In this paper, we propose KALM, a framework that leverages large pre-trained vision-language models (LMs) to automatically generate taskrelevant and cross-instance consistent keypoints. KALM distills robust and consistent keypoints across views and objects by generating proposals using LMs and verifies them against a small set of robot demonstration data. Based on the generated keypoints, we can train keypoint-conditioned policy models that predict actions in keypoint-centric frames, enabling robots to generalize effectively across varying object poses, camera views, and object instances with similar functional shapes. Our method demonstrates strong performance in the real world, adapting to different tasks and environments from only a handful of demonstrations while requiring no additional labels. 

   more » « less
5. Modeling and Analysis of a Soft Endoluminal Inchworm Robot Propelled by a Rotating Magnetic Dipole Field

   https://doi.org/https://doi.org/10.1115/1.4053114  

   Jake A. Steiner, L. N. (October 2022, Journal of mechanisms and robotics)

   In clinical practice, therapeutic and diagnostic endoluminal procedures of the human body often use a scope, catheter, or passive pill-shaped camera. Unfortunately, such devices used in the circulatory system and gastrointestinal tract are often uncomfortable, invasive, and require the patient to be sedated. With current technology, regions of the body are often inaccessible to the clinician. Herein, a magnetically actuated soft endoluminal inchworm robot that may extend clinicians’ ability to reach further into the human body and practice new procedures is described, modeled, and analyzed. A detailed locomotion model is pro- posed that takes into account the elastic deformation of the robot and its interactions with the environment. The model is validated with in vitro and ex vivo (pig intestine) physical experiments and is shown to capture the robot’s gait characteristics through a lumen. Utilizing dimensional analysis, the effects of the mechanical properties and design variables on the robot’s motion are investigated further to advance the understanding of this endoluminal robot concept. 

   more » « less