Search for: All records

Self-supervised and language-supervised image models contain rich knowledge of the world that is important for generalization. Many robotic tasks, however, require a detailed understanding of 3D geometry, which is often lacking in 2D image features. This work bridges this 2D-to-3D gap for robotic manip- ulation by leveraging distilled feature fields to combine accurate 3D geometry with rich semantics from 2D foundation models. We present a few-shot learning method for 6-DOF grasping and placing that harnesses these strong spatial and semantic priors to achieve in-the-wild generalization to unseen objects. Using fea- tures distilled from a vision-language model, CLIP, we present a way to designate novel objects for manipulation via free-text natural language, and demonstrate its ability to generalize to unseen expressions and novel categories of objects. Project website: https://f3rm.csail.mit.edu 

A robot’s deployment environment often involves perceptual changes that differ from what it has experienced during training. Standard practices such as data augmentation attempt to bridge this gap by augmenting source images in an effort to extend the support of the training distribution to better cover what the agent might experience at test time. In many cases, however, it is impossible to know test-time distribution- shift a priori, making these schemes infeasible. In this paper, we introduce a general approach, called Invariance through Latent Alignment (ILA), that improves the test-time performance of a visuomotor control policy in deployment environments with unknown perceptual variations. ILA performs unsupervised adaptation at deployment-time by matching the distribution of latent features on the target domain to the agent’s prior experience, without relying on paired data. Although simple, we show that this idea leads to surprising improvements on a variety of challenging adaptation scenarios, including changes in lighting conditions, the content in the scene, and camera poses. We present results on calibrated control benchmarks in simulation—the distractor control suite—and a physical robot under a sim-to-real setup. Video and code available at: https: //invariance-through-latent-alignment.github.io 

Agile maneuvers such as sprinting and high-speed turning in the wild are challenging for legged robots. We present an end-to-end learned controller that achieves record agility for the MIT Mini Cheetah, sustaining speeds up to 3.9 m/s. This system runs and turns fast on natural terrains like grass, ice, and gravel and responds robustly to disturbances. Our controller is a neural network trained in simulation via reinforcement learning and transferred to the real world. The two key components are (i) an adaptive curriculum on velocity commands and (ii) an online system identification strategy for sim-to-real transfer. Videos of the robot’s behaviors are available at https://agility.csail.mit.edu/ .

 

The dissemination of sensors is key to realizing a sustainable, ‘intelligent’ world, where everyday objects and environments are equipped with sensing capabilities to advance the sustainability and quality of our lives—e.g. via smart homes, smart cities, smart healthcare, smart logistics, Industry 4.0, and precision agriculture. The realization of the full potential of these applications critically depends on the availability of easy-to-make, low-cost sensor technologies. Sensors based on printable electronic materials offer the ideal platform: they can be fabricated through simple methods (e.g. printing and coating) and are compatible with high-throughput roll-to-roll processing. Moreover, printable electronic materials often allow the fabrication of sensors on flexible/stretchable/biodegradable substrates, thereby enabling the deployment of sensors in unconventional settings. Fulfilling the promise of printable electronic materials for sensing will require materials and device innovations to enhance their ability to transduce external stimuli—light, ionizing radiation, pressure, strain, force, temperature, gas, vapours, humidity, and other chemical and biological analytes. This Roadmap brings together the viewpoints of experts in various printable sensing materials—and devices thereof—to provide insights into the status and outlook of the field. Alongside recent materials and device innovations, the roadmap discusses the key outstanding challenges pertaining to each printable sensing technology. Finally, the Roadmap points to promising directions to overcome these challenges and thus enable ubiquitous sensing for a sustainable, ‘intelligent’ world.

 

Current model-based reinforcement learning methods struggle when operating from complex visual scenes due to their inability to prioritize task-relevant features. To mitigate this prob- lem, we propose learning Task Informed Ab- stractions (TIA) that explicitly separates reward- correlated visual features from distractors. For learning TIA, we introduce the formalism of Task Informed MDP (TiMDP) that is realized by train- ing two models that learn visual features via coop- erative reconstruction, but one model is adversari- ally dissociated from the reward signal. Empirical evaluation shows that TIA leads to significant per- formance gains over state-of-the-art methods on many visual control tasks where natural and un- constrained visual distractions pose a formidable challenge. Project page: https://xiangfu.co/tia