In this demonstration, in collaboration with licensed therapists, we introduce an AI therapist that takes advantage of the smart-home environment to screen day-to-day functioning and infer mental wellness of an occupant. Our system can assess a user's daily functioning and mental wellness based on a combination of direct conversation with users and information obtained from smart home devices using psychological rubrics proposed in [1]. We demonstrate that our system can converse with a user in a natural way (through a smartphone or smart speaker) and analyze a user's response semantically and sentimentally. In addition, we show that our system can provide preliminary interventions to help improve the user's wellness. In particular, when abnormal behavior is detected during the conversation or by smart home devices, the system provides psychotherapeutic consolations during the conversation and will check on the occupant's condition by actuating a home robot.
more »
« less
Conversational AI Therapist for Daily Function Screening in Home Environments
The growth of smart devices is making typical homes more intelligent. In this work, in collaboration with therapists, we introduce a home-based AI therapist that takes advantage of the smart home environment to screen the day-to-day functioning and infer mental wellness of an occupant. Unlike existing “chatbot” works that identify the mental status of users through conversation, our AI therapist additionally leverages smart devices and sensors throughout the home to infer mental well-being and assesses a user's daily functioning. We propose a series of 37 dimensions of daily functioning, that our system observes through conversing with the user and detecting daily activity events using sensors and smart sensors throughout the home. Our system utilizes these 37 dimensions in conjunction with novel natural language processing architectures to detect abnormalities in mental status (e.g., angry or depressed), well-being, and daily functioning and generate responses to console users when abnormalities are detected. Through a series of user studies, we demonstrate that our system can converse with a user naturally, accurately detect abnormalities in well-being, and provide appropriate responses consoling users.
more »
« less
- Award ID(s):
- 1815274
- PAR ID:
- 10416029
- Date Published:
- Journal Name:
- IASA '22: Proceedings of the 1st ACM International Workshop on Intelligent Acoustic Systems and Applications
- Page Range / eLocation ID:
- 31 to 36
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Recent advances in cyber-physical systems, artificial intelligence, and cloud computing have driven the wide deployments of Internet-of-things (IoT) in smart homes. As IoT devices often directly interact with the users and environments, this paper studies if and how we could explore the collective insights from multiple heterogeneous IoT devices to infer user activities for home safety monitoring and assisted living. Specifically, we develop a new system, namely IoTMosaic, to first profile diverse user activities with distinct IoT device event sequences, which are extracted from smart home network traffic based on their TCP/IP data packet signatures. Given the challenges of missing and out-of-order IoT device events due to device malfunctions or varying network and system latencies, IoTMosaic further develops simple yet effective approximate matching algorithms to identify user activities from real-world IoT network traffic. Our experimental results on thousands of user activities in the smart home environment over two months show that our proposed algorithms can infer different user activities from IoT network traffic in smart homes with the overall accuracy, precision, and recall of 0.99, 0.99, and 1.00, respectively.more » « less
-
Smart-home devices have become integral to daily routines, but their onboarding procedures - setting up a newly acquired smart device into operational mode - remain understudied. The heterogeneity of smart-home devices and their onboarding procedure can easily overwhelm users when they scale up their smart-home system. While Matter, the new IoT standard, aims to unify the smart-home ecosystem, it is still evolving, resulting in mixed compliance among devices. In this paper, we study the complexity of device onboarding from users' perspectives. We thus performed cognitive walkthroughs on 12 commercially available smart-home devices, documenting the commonality and distinctions of the onboarding process across these devices. We found that onboarding smart home devices can often be tedious and confusing. Users must devote significant time to creating an account, searching for the target device, and providing Wi-Fi credentials for each device they install. Matter-compatible devices are supposedly easier to manage, as they can be registered through one single hub independent of the vendor. Unfortunately, we found such a statement is not always true. Some devices still need their own companion apps and accounts to fully function. Based on our observations, we give recommendations about how to support a more user-friendly onboarding process.more » « less
-
We proposed a Client Server Based Plastic Fiber Optic Network for an Elderly Support system in a smart home environment. The network will incorporate multimodal dialogue systems (MDS) and Artificial Intelligence (AI) systems. Existing AI models like GPT and Vicuna will be used and further trained to support the elderly in a smart home. The MDS systems, Whisper and Lavis, will act as intermediaries between the POF network of sensors and the AI systems. The systems will convert video and audio files into information that the AI systems can process and respond to. The POF network, consisting of sensors and a client-server architecture, will serve as the system's backbone. Its primary purpose is to gather data for the AI system and act based on its output. This research aims to enhance the safety, well-being, and independence of the elderly by leveraging advanced network technologies.more » « less
-
null (Ed.)User authentication is a critical process in both corporate and home environments due to the ever-growing security and privacy concerns. With the advancement of smart cities and home environments, the concept of user authentication is evolved with a broader implication by not only preventing unauthorized users from accessing confidential information but also providing the opportunities for customized services corresponding to a specific user. Traditional approaches of user authentication either require specialized device installation or inconvenient wearable sensor attachment. This article supports the extended concept of user authentication with a device-free approach by leveraging the prevalent WiFi signals made available by IoT devices, such as smart refrigerator, smart TV, and smart thermostat, and so on. The proposed system utilizes the WiFi signals to capture unique human physiological and behavioral characteristics inherited from their daily activities, including both walking and stationary ones. Particularly, we extract representative features from channel state information (CSI) measurements of WiFi signals, and develop a deep-learning-based user authentication scheme to accurately identify each individual user. To mitigate the signal distortion caused by surrounding people’s movements, our deep learning model exploits a CNN-based architecture that constructively combines features from multiple receiving antennas and derives more reliable feature abstractions. Furthermore, a transfer-learning-based mechanism is developed to reduce the training cost for new users and environments. Extensive experiments in various indoor environments are conducted to demonstrate the effectiveness of the proposed authentication system. In particular, our system can achieve over 94% authentication accuracy with 11 subjects through different activities.more » « less