More Like this

1. A Method for Constraint Inference Using Pose and Wrench Measurements

   https://doi.org/arXiv:2010.15916  

   Subramani, Guru; Hagenow, Michael; Gleicher, Michael; Zinn, Michael (October 2020, ArXivorg)

   null (Ed.)

   Many physical tasks such as pulling out a drawer or wiping a table can be modeled with geometric constraints. These geometric constraints are characterized by restrictions on kinematic trajectories and reaction wrenches (forces and moments) of objects under the influence of the constraint. This paper presents a method to infer geometric constraints involving unmodeled objects in human demonstrations using both kinematic and wrench measurements. Our approach takes a recording of a human demonstration and determines what constraints are present, when they occur, and their parameters (e.g. positions). By using both kinematic and wrench information, our methods are able to reliably identify a variety of constraint types, even if the constraints only exist for short durations within the demonstration. We present a systematic approach to fitting arbitrary scleronomic constraint models to kinematic and wrench measurements. Reaction forces are estimated from measurements by removing friction. Position, orientation, force, and moment error metrics are developed to provide systematic comparison between constraint models. By conducting a user study, we show that our methods can reliably identify constraints in realistic situations and confirm the value of including forces and moments in the model regression and selection process. 

   more » « less
2. LeTO: Learning Constrained Visuomotor Policy With Differentiable Trajectory Optimization

   https://doi.org/10.1109/TASE.2024.3486542  

   Xu, Zhengtong; She, Yu (March 2025, IEEE Transactions on Automation Science and Engineering)

   This paper introduces LeTO, a method for learning constrained visuomotor policy with differentiable trajectory optimization. Our approach integrates a differentiable optimization layer into the neural network. By formulating the optimization layer as a trajectory optimization problem, we enable the model to end-to-end generate actions in a safe and constraint-controlled fashion without extra modules. Our method allows for the introduction of constraint information during the training process, thereby balancing the training objectives of satisfying constraints, smoothing the trajectories, and minimizing errors with demonstrations. This “gray box” method marries optimization-based safety and interpretability with powerful representational abilities of neural networks. We quantitatively evaluate LeTO in simulation and in the real robot. The results demonstrate that LeTO performs well in both simulated and real-world tasks. In addition, it is capable of generating trajectories that are less uncertain, higher quality, and smoother compared to existing imitation learning methods. Therefore, it is shown that LeTO provides a practical example of how to achieve the integration of neural networks with trajectory optimization. We release our code at https://github.com/ZhengtongXu/LeTO. 

   more » « less
3. Visual Inverse Kinematics: Finding Feasible Robot Poses under Kinematic and Vision Constraints

   Wu, Liangting; Tron, Roberto (July 2025, W Telekomunikacji)

   We introduce Visual Inverse Kinematics (VIK), which finds kinematically feasible joint configurations that satisfy vision-based constraints, bridging the gap between inverse kinematics (IK) and visual servoing (VS). Unlike IK, no explicit end-effector pose is given, and unlike VS, exact image measurements may not be available. In this work, we develop a formulation of the VIK problem with a field of view (FoV) constraint, enforcing the visibility of an object from a camera on the robot. Our proposed solution introduces a virtual kinematic chain that connects the physical robot and the object, transforming the FoV constraint into a joint angle kinematic constraint. Along the way, we introduce multiple vision-based cost functions to fulfill different objectives. We solve this formulation of the VIK problem using a method that involves a semidefinite program (SDP) constraint followed by a rank minimization algorithm. The performance of this method for solving the VIK problem is validated through simulations. 

   more » « less
4. Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videos

   Chang, Matthew; Prakash, Aditya; Gupta, Saurabh (December 2023, Neural Information Processing Systems)

   The analysis and use of egocentric videos for robotic tasks is made challenging by occlusion due to the hand and the visual mismatch between the human hand and a robot end-effector. In this sense, the human hand presents a nuisance. However, often hands also provide a valuable signal, e.g. the hand pose may suggest what kind of object is being held. In this work, we propose to extract a factored representation of the scene that separates the agent (human hand) and the environment. This alleviates both occlusion and mismatch while preserving the signal, thereby easing the design of models for downstream robotics tasks. At the heart of this factorization is our proposed Video Inpainting via Diffusion Model (VIDM) that leverages both a prior on real-world images (through a large-scale pre-trained diffusion model) and the appearance of the object in earlier frames of the video (through attention). Our experiments demonstrate the effectiveness of VIDM at improving inpainting quality on egocentric videos and the power of our factored representation for numerous tasks: object detection, 3D reconstruction of manipulated objects, and learning of reward functions, policies, and affordances from videos. 

   more » « less
5. Combining and Decoupling Rigid and Soft Grippers to Enhance Robotic Manipulation

   https://doi.org/10.1089/soro.2024.0062  

   Keely, Maya; Kim, Yeunhee; Mehta, Shaunak A; Hoegerman, Joshua; Ramirez_Sanchez, Robert; Paul, Emily; Mills, Camryn; Losey, Dylan P; Bartlett, Michael D (March 2025, Soft Robotics)

   For robot arms to perform everyday tasks in unstructured environments, these robots must be able to manipulate a diverse range of objects. Today’s robots often grasp objects with either soft grippers or rigid end-effectors. However, purely rigid or purely soft grippers have fundamental limitations as follows: soft grippers struggle with irregular heavy objects, whereas rigid grippers often cannot grasp small numerous items. In this article, we therefore introduce RISOs, a mechanics and controls approach for unifying traditional RIgid end-effectors with a novel class of SOft adhesives. When grasping an object, RISOs can use either the rigid end-effector (pinching the item between nondeformable fingers) and/or the soft materials (attaching and releasing items with switchable adhesives). This enhances manipulation capabilities by combining and decoupling rigid and soft mechanisms. With RISOs, robots can perform grasps along a spectrum from fully rigid, to fully soft, to rigid-soft, enabling real-time object manipulation across a 1.5 million times range in weight (from 2 mg to 2.9 kg). To develop RISOs, we first model and characterize the soft switchable adhesives. We then mount sheets of these soft adhesives on the surfaces of rigid end-effectors and develop control strategies that make it easier for robot arms and human operators to utilize RISOs. The resulting RISO grippers were able to pick up, carry, and release a larger set of objects than existing grippers, and participants also preferred using RISO. Overall, our experimental and user study results suggest that RISOs provide an exceptional gripper range in both capacity and object diversity. 

   more » « less