Search for: All records

Foundation models (FM) have shown immense human-like capabilities for generating digital media. However, foundation models that can freely sense, interact, and actuate the physical domain is far from being realized. This is due to 1) requiring dense deployments of sensors to fully cover and analyze large spaces, while 2) events often being localized to small areas, making it difficult for FMs to pinpoint relevant areas of interest relevant to the current task. We propose FlexiFly, a platform that enables FMs to “zoom in” and analyze relevant areas with higher granularity to better understand the physical environment and carry out tasks. FlexiFly accomplishes by introducing 1) a novel image segmentation technique that aids in identifying relevant locations and 2) a modular and reconfigurable sensing and actuation drone platform that FMs can actuate to “zoom in” with relevant sensors and actuators. We demonstrate through real smart home deployments that FlexiFly enables FMs and LLMs to complete diverse tasks up to 85% more successfully. FlexiFly is critical step towards FMs and LLMs that can naturally interface with the physical world. 

Foundation models excel in tasks such as content generation, zero-shot classifications, and reasoning. However, they struggle with sensing, interacting, and actuating in the physical world due to their dependence on limited sensors and actuators in providing timely contextual information or physical interactions. This reliance restricts the system’s adaptability and coverage. To address these issues and create an embodied AI with foundation models (FMs), we introduce Embodied Reconfigurable Drone Agent (EmbodiedRDA). EmbodiedRDA features a custom drone platform that can autonomously swap payloads to reconfigure itself with a diverse list of sensors and actuators. We designed FM agents to instruct the drone to equip itself with appropriate physical modules, analyze sensor data, make decisions, and control the drone’s actions. This enables the system to perform a variety of tasks in dynamic physical environments, bridging the gap between the digital and physical worlds. 

Recent research has investigated the importance of both walkable urban design and social cohesion. Social cohesion has been shown to have broad social and health benefits, and scholars have hypothesized that walkable urban design can influence cohesion, though evidence remains limited. In this work, we leveraged a data-driven approach that broke down design factors related to walkable design and investigated their impact on cohesion. We used a US-wide open urban form dataset to characterize walkable urban design, and we used an open survey dataset that measured cohesion and demographics with a total sample size of 9670 in six US cities. We leveraged partial least squared structural equation modeling for statistical analysis. We found, controlling for demographics, that land use diversity had a significant positive impact on social cohesion. We also found that physical density, social density, and transit connectedness had significant negative impacts on cohesion, though this association is largely driven by the very dense neighborhoods in cities. These findings shed light on different theories of the built environment, offering insights for designers, engineers, and policymakers interested in the social effects of the built environment. 

We present the design and implementation of RECA, a novel human-centric recommender system for co-optimizing energy consumption, comfort and air quality in commercial buildings. Existing works generally optimize these objectives separately, or by only controlling energy consuming resources within the building without directly engaging occupants. We develop a deep reinforcement learning architecture based on multitask learning, demonstrate how it can be used to jointly learn energy savings, comfort and air quality improvements for different actions, and build a recommender system with humans-in-the-loop. Through real deployments in multiple commercial buildings, we found that RECA has the potential to further reduce energy consumption by up to in energy-focused optimization, improve all objectives by in joint optimization, and improve thermal comfort by up to in comfort and air quality focused optimization, over existing solutions. 

Mapping 3D airflow fields is important for many HVAC, industrial, medical, and home applications. However, current approaches are expensive and time-consuming. We present Anemoi, a sub-$100 drone-based system for autonomously mapping 3D airflow fields in indoor environments. Anemoi leverages the effects of airflow on motor control signals to estimate the magnitude and direction of wind at any given point in space. We introduce an exploration algorithm for selecting optimal waypoints that minimize overall airflow estimation uncertainty. We demonstrate through microbenchmarks and real deployments that Anemoi is able to estimate wind speed and direction with errors up to 0.41 m/s and 25.1° lower than the existing state of the art and map 3D airflow fields with an average RMS error of 0.73 m/s. 

Domain-specific sensor deployments are critical to enabling various IoT applications. Existing solutions for quickly deploying sensing systems require significant amount of work and time, even for experienced engineers. We propose LegoSENSE, a low-cost open-source and modular platform, built on top of the widely popular Raspberry Pi single-board computer, that makes it simple for anyone to rapidly set up and deploy a customized sensing solution for application specific IoT deployments. In addition, the ‘plug and play’ and ‘mix and match’ functionality of LegoSENSE makes the sensor modules reusable, and allows them to be mixed and matched to serve a variety of needs. We show, through a series of user studies, that LegoSENSE enables users without engineering background to deploy a wide range of applications up to 9 × faster than experienced engineers without the use of LegoSENSE. We open-source the hardware and software designs to foster an ever-evolving community, enabling IoT applications for enthusiasts, students, scientists, and researchers across various application domains with or without prior experiences with embedded platforms or coding. 

The stethoscope is one of the most important diagnostic tools used by healthcare professionals, through a process called auscultation, to screen patients for abnormalities of the heart and lungs. While there are digital stethoscopes on the market which ease this process, it still takes years of training to properly use these devices to listen for abnormal sounds within the body. We present ARSteth, an intelligent stethoscope platform that improves the accessibility of stethoscopes for the general population, allowing anyone to perform auscultation in the comfort of their own homes. Our platform utilizes a combination of augmented reality (AR), acoustic intelligence, and human-machine interaction to dynamically guide users on where to place the stethoscope on different parts of the body (auscultation points), through visual and audio cues. Through user studies, we show that ARSteth, on average, can guide users within 13.2 mm from optimal auscultation points marked by licensed physicians in 13.09 seconds for each auscultation point. By guiding users towards more effective auscultation points, make preventative health screening more accessible and effective for everyone we are able to achieve higher confidence on classifying heart murmurs. 

Cardiopulmonary ailments are a major cause of mortality. Stethoscopes are one of the most important tools that healthcare professionals use to screen patients for a variety of ailments, especially those related to the heart and lungs. Despite the growth of digital stethoscopes on the market, it takes years of training to properly use stethoscopes to listen for abnormal sounds within the body. In this demonstration, we present an intelligent stethoscope platform that makes stethoscopes more accessible to the general population. Our platform utilizes augmented reality (AR) to provide real-time guidance on where to properly place the stethoscope on the body, enabling the general population to screen themselves for ailments. 

With the global spread of the COVID-19 pandemic, ventilation indoors is becoming increasingly important in preventing the spread of airborne viruses. However, while sensors exist to measure wind speed and airflow gradients, they must be manually held by a human or an autonomous vehicle, robot, or drone that moves around the space to build an airflow map of the environment. In this demonstration, we present DAE, a novel drone-based system that can automatically navigate and estimate air flow in a space without the need of additional sensors attached onto the drone. DAE directly utilizes the flight controller data that all drones use to self-stabilize in the air to estimate airflow. DAE estimates airflow gradients in a room based on how the flight controller adjusts the motors on the drone to compensate external perturbations and air currents, without the need for attaching additional wind or airflow sensors. 

Recent years have witnessed the increasing penetration of wireless charging base stations in the workplace and public areas, such as airports and cafeterias. Such an emerging wireless charging infrastructure has presented opportunities for new indoor localization and identification services for mobile users. In this paper, we present QID, the first system that can identify a Qi-compliant mobile device during wireless charging in real-time. QID extracts features from the clock oscillator and control scheme of the power receiver and employs light-weight algorithms to classify the device. QID adopts a 2-dimensional motion unit to emulate a variety of multi-coil designs of Qi, which allows for fine-grained device fingerprinting. Our results show that QID achieves high recognition accuracy. With the prevalence of public wireless charging stations, our results also have important implications for mobile user privacy.