skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Attention:

The NSF Public Access Repository (PAR) system and access will be unavailable from 10:00 PM to 12:00 PM ET on Tuesday, March 25 due to maintenance. We apologize for the inconvenience.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Conversational AI Therapist for Daily Function Screening in Home Environments
The growth of smart devices is making typical homes more intelligent. In this work, in collaboration with therapists, we introduce a home-based AI therapist that takes advantage of the smart home environment to screen the day-to-day functioning and infer mental wellness of an occupant. Unlike existing “chatbot” works that identify the mental status of users through conversation, our AI therapist additionally leverages smart devices and sensors throughout the home to infer mental well-being and assesses a user's daily functioning. We propose a series of 37 dimensions of daily functioning, that our system observes through conversing with the user and detecting daily activity events using sensors and smart sensors throughout the home. Our system utilizes these 37 dimensions in conjunction with novel natural language processing architectures to detect abnormalities in mental status (e.g., angry or depressed), well-being, and daily functioning and generate responses to console users when abnormalities are detected. Through a series of user studies, we demonstrate that our system can converse with a user naturally, accurately detect abnormalities in well-being, and provide appropriate responses consoling users.  more » « less
Award ID(s):
1815274
PAR ID:
10416029
Author(s) / Creator(s):
; ; ; ; ;
Date Published:
Journal Name:
IASA '22: Proceedings of the 1st ACM International Workshop on Intelligent Acoustic Systems and Applications
Page Range / eLocation ID:
31 to 36
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; ; ; ; (, SenSys '22: Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems)
    In this demonstration, in collaboration with licensed therapists, we introduce an AI therapist that takes advantage of the smart-home environment to screen day-to-day functioning and infer mental wellness of an occupant. Our system can assess a user's daily functioning and mental wellness based on a combination of direct conversation with users and information obtained from smart home devices using psychological rubrics proposed in [1]. We demonstrate that our system can converse with a user in a natural way (through a smartphone or smart speaker) and analyze a user's response semantically and sentimentally. In addition, we show that our system can provide preliminary interventions to help improve the user's wellness. In particular, when abnormal behavior is detected during the conversation or by smart home devices, the system provides psychotherapeutic consolations during the conversation and will check on the occupant's condition by actuating a home robot. 
    more » « less
  2. ; ; ; ; (, ACM)
    Smart-home devices have become integral to daily routines, but their onboarding procedures - setting up a newly acquired smart device into operational mode - remain understudied. The heterogeneity of smart-home devices and their onboarding procedure can easily overwhelm users when they scale up their smart-home system. While Matter, the new IoT standard, aims to unify the smart-home ecosystem, it is still evolving, resulting in mixed compliance among devices. In this paper, we study the complexity of device onboarding from users' perspectives. We thus performed cognitive walkthroughs on 12 commercially available smart-home devices, documenting the commonality and distinctions of the onboarding process across these devices. We found that onboarding smart home devices can often be tedious and confusing. Users must devote significant time to creating an account, searching for the target device, and providing Wi-Fi credentials for each device they install. Matter-compatible devices are supposedly easier to manage, as they can be registered through one single hub independent of the vendor. Unfortunately, we found such a statement is not always true. Some devices still need their own companion apps and accounts to fully function. Based on our observations, we give recommendations about how to support a more user-friendly onboarding process. 
    more » « less
  3. ; ; ; ; ; ; (, Proceedings of the 2021 IEEE Conference on Smart Computing (SmartComp))
    null (Ed.)
    Smart-home devices promise to make users’ lives more convenient. However, at the same time, such devices increase the possibility of breaching users’ privacy as they are tightly connected to the users’ daily lives and activities. To address privacy invasion through smart-home devices, we present ChatterHub. This novel approach accurately identifies smart-home devices’ activities with minimal monitoring of encrypted traffic in the home network. ChatterHub targets devices that can only connect to the Internet through a centralized smart-home hub (e.g., Samsung SmartThings) using Zigbee or Z-wave. Specifically, ChatterHub passively eavesdrops on encrypted network traffic from the hub and leverages machine learning techniques to classify events and states of smart-home devices. Using ChatterHub, an adversary can identify smart-home devices’ specific activities without prior knowledge of the target smart home (e.g., list of deployed devices, types of communication protocols). We evaluated the accuracy and efficiency of ChatterHub in three real-world smart-home environments, and the evaluation results show that an attacker can successfully disclose smart-home devices’ behaviors with over 88% F1 score. We further demonstrate that ChatterHub successfully recognizes privacy-sensitive activities, including open and close of a smart door lock and turn on and off of smart LED. Additionally, to mitigate the threats posed by ChatterHub, we introduce two approaches, packet padding and random sequence injection. These mitigation approaches can effectively prevent threats from ChatterHub with only 9.2MB of additional network traffic per day. 
    more » « less
  4. ; ; ; (, IEEE INFOCOM2022)
    Recent advances in cyber-physical systems, artificial intelligence, and cloud computing have driven the wide deployments of Internet-of-things (IoT) in smart homes. As IoT devices often directly interact with the users and environments, this paper studies if and how we could explore the collective insights from multiple heterogeneous IoT devices to infer user activities for home safety monitoring and assisted living. Specifically, we develop a new system, namely IoTMosaic, to first profile diverse user activities with distinct IoT device event sequences, which are extracted from smart home network traffic based on their TCP/IP data packet signatures. Given the challenges of missing and out-of-order IoT device events due to device malfunctions or varying network and system latencies, IoTMosaic further develops simple yet effective approximate matching algorithms to identify user activities from real-world IoT network traffic. Our experimental results on thousands of user activities in the smart home environment over two months show that our proposed algorithms can infer different user activities from IoT network traffic in smart homes with the overall accuracy, precision, and recall of 0.99, 0.99, and 1.00, respectively. 
    more » « less
  5. ; ; (, 2021 22nd International Symposium on Quality Electronic Design (ISQED))
    null (Ed.)
    Anxiety disorders affect more than 18 percent of the population and is the most common mental illnesses in the US. There is a great demand to address this emerging epidemic with tools to differentiate and diagnose such disorders, and to create awareness especially in places like NorthEast Texas which is home to 1.5 million people with 58 percent of them living in rural areas. The goal of the proposed device is to diagnose as many anxiety disorders as possible, in real-time using the diagnosing wearable framework, SolicitudeSavvy, which uses technology such as the Internet of Things (IoT), a network of interconnected devices, to accomplish such a task. The proposed IoT-based device has two components: a custom-built wearable necklace that contains sensors to collect data about the user as they go about their day and a low-cost portable system that monitors Electrooculography (EoG) signals using a camera. The partial necklace attaches to the shirt and opens halfway around the wearer's neck and the EoG can be attached to any eyewear. The device monitors the user throughout the day, and even as they go to bed at night. This information is accumulated in the IoT cloud and analyzed to see exactly what type of disorder(s) the patient may suffer from. The authorized personnel i.e. doctor or therapist can use this pattern to find a treatment that best suits them and is most likely to resolve their affliction. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email