Search for: All records

Here's the cleaned-up version: As micro-drones become increasingly deployed in indoor environments for applications ranging from warehouse inspection to emergency response, the challenge of precise automated landing emerges as a crucial barrier to their practical operation and ubiquitous adoption. Existing landing approaches often require complex hardware and substantial computation or perform unreliably indoors, making them impractical for palm-sized microdrones. We propose Moth, a low-cost infrared light-based solution that targets precise and efficient landing of low-resource microdrones. Moth consists of an infrared light source at the landing station along with an energy-efficient photodiode (PD) sensing platform attached to the bottom of the drone. At a cost under 83 USD, Moth achieves comparable performance to vision-based methods but at a fraction of the energy consumption and computation. Moth requires only three PDs without any complex pattern recognition models to land the drone accurately, under 10 cm of error, from up to 11.1 m away. 

Large Language Models (LLMs) have shown immense human-like capabilities for reasoning and generating digital content. However, their ability to freely sense, interact, and actuate the physical domain remains significantly limited due to three fundamental challenges: (1) physical environments require specialized sensors for different tasks, yet deploying dedicated sensors for each application is impractical; (2) events and objects of interest are often localized to small areas within large spaces, making them difficult to detect with static sensor networks; and (3) foundation models need flexible actuation capabilities to meaningfully interact with the physical world. To bridge this gap, we introduce EmbodiedFly, an embodied LLM agent combining a foundation model pipeline with a reconfigurable drone platform to observe, understand, and interact with the physical world. Our co-design approach features (1) a FM orchestration framework connecting multiple LLMs, VLMs, and an open-set object detection model; (2) a novel image segmentation technique that identifies task-relevant areas; and (3) a custom drone platform that autonomously reconfigures with appropriate sensors and actuators based on commands from the FM orchestration framework. Through real-world deployments, we demonstrate that EmbodiedFly completes diverse physical tasks with up to 85% higher success rates compared to traditional approaches leveraging static deployments. 

Foundation models (FM) have shown immense human-like capabilities for generating digital media. However, foundation models that can freely sense, interact, and actuate the physical domain is far from being realized. This is due to 1) requiring dense deployments of sensors to fully cover and analyze large spaces, while 2) events often being localized to small areas, making it difficult for FMs to pinpoint relevant areas of interest relevant to the current task. We propose FlexiFly, a platform that enables FMs to “zoom in” and analyze relevant areas with higher granularity to better understand the physical environment and carry out tasks. FlexiFly accomplishes by introducing 1) a novel image segmentation technique that aids in identifying relevant locations and 2) a modular and reconfigurable sensing and actuation drone platform that FMs can actuate to “zoom in” with relevant sensors and actuators. We demonstrate through real smart home deployments that FlexiFly enables FMs and LLMs to complete diverse tasks up to 85% more successfully. FlexiFly is critical step towards FMs and LLMs that can naturally interface with the physical world. 

Foundation models excel in tasks such as content generation, zero-shot classifications, and reasoning. However, they struggle with sensing, interacting, and actuating in the physical world due to their dependence on limited sensors and actuators in providing timely contextual information or physical interactions. This reliance restricts the system’s adaptability and coverage. To address these issues and create an embodied AI with foundation models (FMs), we introduce Embodied Reconfigurable Drone Agent (EmbodiedRDA). EmbodiedRDA features a custom drone platform that can autonomously swap payloads to reconfigure itself with a diverse list of sensors and actuators. We designed FM agents to instruct the drone to equip itself with appropriate physical modules, analyze sensor data, make decisions, and control the drone’s actions. This enables the system to perform a variety of tasks in dynamic physical environments, bridging the gap between the digital and physical worlds. 

The stethoscope is one of the most important diagnostic tools used by healthcare professionals, through a process called auscultation, to screen patients for abnormalities of the heart and lungs. While there are digital stethoscopes on the market which ease this process, it still takes years of training to properly use these devices to listen for abnormal sounds within the body. We present ARSteth, an intelligent stethoscope platform that improves the accessibility of stethoscopes for the general population, allowing anyone to perform auscultation in the comfort of their own homes. Our platform utilizes a combination of augmented reality (AR), acoustic intelligence, and human-machine interaction to dynamically guide users on where to place the stethoscope on different parts of the body (auscultation points), through visual and audio cues. Through user studies, we show that ARSteth, on average, can guide users within 13.2 mm from optimal auscultation points marked by licensed physicians in 13.09 seconds for each auscultation point. By guiding users towards more effective auscultation points, make preventative health screening more accessible and effective for everyone we are able to achieve higher confidence on classifying heart murmurs. 

Mapping 3D airflow fields is important for many HVAC, industrial, medical, and home applications. However, current approaches are expensive and time-consuming. We present Anemoi, a sub-$100 drone-based system for autonomously mapping 3D airflow fields in indoor environments. Anemoi leverages the effects of airflow on motor control signals to estimate the magnitude and direction of wind at any given point in space. We introduce an exploration algorithm for selecting optimal waypoints that minimize overall airflow estimation uncertainty. We demonstrate through microbenchmarks and real deployments that Anemoi is able to estimate wind speed and direction with errors up to 0.41 m/s and 25.1° lower than the existing state of the art and map 3D airflow fields with an average RMS error of 0.73 m/s. 

Domain-specific sensor deployments are critical to enabling various IoT applications. Existing solutions for quickly deploying sensing systems require significant amount of work and time, even for experienced engineers. We propose LegoSENSE, a low-cost open-source and modular platform, built on top of the widely popular Raspberry Pi single-board computer, that makes it simple for anyone to rapidly set up and deploy a customized sensing solution for application specific IoT deployments. In addition, the ‘plug and play’ and ‘mix and match’ functionality of LegoSENSE makes the sensor modules reusable, and allows them to be mixed and matched to serve a variety of needs. We show, through a series of user studies, that LegoSENSE enables users without engineering background to deploy a wide range of applications up to 9 × faster than experienced engineers without the use of LegoSENSE. We open-source the hardware and software designs to foster an ever-evolving community, enabling IoT applications for enthusiasts, students, scientists, and researchers across various application domains with or without prior experiences with embedded platforms or coding. 

Breath monitoring is important for monitoring illnesses, such as sleep apnea, for people of all ages. One cause of concern for parents is sudden infant death syndrome (SIDS), where an infant suddenly passes away during sleep, usually due to complications in breathing. There are a variety of works and products on the market for monitoring breathing, especially for children and infants. Many of these are wearables that require you to attach an accessory onto the child or person, which can be uncomfortable. Other solutions utilize a camera, which can be privacy-intrusive and function poorly during the night, when lighting is poor. In this work, we introduce BuMA, an audio-based, non-intrusive, and contactless, breathing monitoring system. BuMA utilizes a microphone array, beamforming, and audio filtering to enhance the sounds of breathing by filtering out several common noises in or near home environments, such as construction, speech, and music, that could make detection difficult. We show that BuMA improves breathing detection accuracy by up to 12%, within 30cm from a person, over existing audio filtering algorithms or platforms that do not leverage filtering. 

Cardiopulmonary ailments are a major cause of mortality. Stethoscopes are one of the most important tools that healthcare professionals use to screen patients for a variety of ailments, especially those related to the heart and lungs. Despite the growth of digital stethoscopes on the market, it takes years of training to properly use stethoscopes to listen for abnormal sounds within the body. In this demonstration, we present an intelligent stethoscope platform that makes stethoscopes more accessible to the general population. Our platform utilizes augmented reality (AR) to provide real-time guidance on where to properly place the stethoscope on the body, enabling the general population to screen themselves for ailments. 

There has been an immense growth in sensors, actuators, and smart devices in recent years, which enable us to better sense, actuate, and understand the physical world. Despite this growth, we have yet to achieve fully intelligent environments. This is, in part, due to the large number of different organizations creating smart devices with proprietary technologies and communication protocols that are not compatible with each other and require significant engineering to incorporate and adapt to specific applications. In this work, we present an easy-to-install and low-cost embedded platform that allows users to rapidly configure a mixture of sensors and actuators. The system is based on the commonly-used Raspberry Pi ecosystem, easily configurable, and does not require users to have prior knowledge of programming, which allows anyone, regardless of background, to use. We also introduce a battery-powered wireless extension module that is suitable for mobile drone applications, where a chord-powered Raspberry Pi is not suitable. We demonstrate the impact our system has on enabling drones with flexible sensing modalities and creating smarter environments by integrating our platform into a variety of intelligent home applications.