Search for: All records

We survey applications of pretrained foundation models in robotics. Traditional deep learning models in robotics are trained on small datasets tailored for specific tasks, which limits their adaptability across diverse applications. In contrast, foundation models pretrained on internet-scale data appear to have superior generalization capabilities, and in some instances display an emergent ability to find zero-shot solutions to problems that are not present in the training data. Foundation models may hold the potential to enhance various components of the robot autonomy stack, from perception to decision-making and control. For example, large language models can generate code or provide common sense reasoning, while vision-language models enable open-vocabulary visual recognition. However, significant open research challenges remain, particularly around the scarcity of robot-relevant training data, safety guarantees and uncertainty quantification, and real-time execution. In this survey, we study recent papers that have used or built foundation models to solve robotics problems. We explore how foundation models contribute to improving robot capabilities in the domains of perception, decision-making, and control. We discuss the challenges hindering the adoption of foundation models in robot autonomy and provide opportunities and potential pathways for future advancements. The GitHub project corresponding to this paper can be found here: https://github.com/robotics-survey/Awesome-Robotics-Foundation-Models .

 

Autonomous survey and aerial photogrammetry applications require solving a path planning problem that ensures sensor coverage over a specified area. In this work, we provide a multi-robot path planning method that can obtain this coverage over an arbitrary area of interest. We extend our previous method, path optimization for population counting with overhead robotic networks (POPCORN), by a divide-and-conquer scheme, split and link tiles (SALT), which drastically decreases the time needed for route planning. These POPCORN instances can be computed in parallel and combined with SALT in a scalable manner to produce coverage paths over very large areas of interest. To demonstrate this algorithm’s capabilities, we implemented our planning algorithm with a team of drones to conduct multiple photographic aerial wildlife surveys of the Cape Crozier Adélie penguin rookery on Ross Island, Antarctica, one of the largest Adélie penguin colonies in the world. The colony, which contains over 300,000 nesting pairs and spans over 2 km, was surveyed in about 3 hours. In contrast, previous human-piloted single-drone surveys of the same colony required over 2 days to complete. We also have deployed our survey system at several islets at Mono Lake, California, to survey a California gull colony as well as at a 2000-acre ranch in Marin, California. We provide this survey path planning tool as an open-source software package named wadl.