Speech as a natural and low-burden input modality has great potential to support personal data capture. However, little is known about how people use speech input, together with traditional touch input, to capture different types of data in self-tracking contexts. In this work, we designed and developed NoteWordy, a multimodal self-tracking application integrating touch and speech input, and deployed it in the context of productivity tracking for two weeks (N = 17). Our participants used the two input modalities differently, depending on the data type as well as personal preferences, error tolerance for speech recognition issues, and social surroundings. Additionally, we found speech input reduced participants' diary entry time and enhanced the data richness of the free-form text. Drawing from the findings, we discuss opportunities for supporting efficient personal data capture with multimodal input and implications for improving the user experience with natural language input to capture various self-tracking data.
more »
« less
Data@Hand: Fostering Visual Exploration of Personal Data on Smartphones Leveraging Speech and Touch Interaction
Most mobile health apps employ data visualization to help people view their health and activity data, but these apps provide limited support for visual data exploration. Furthermore, despite its huge potential benefits, mobile visualization research in the personal data context is sparse. This work aims to empower people to easily navigate and compare their personal health data on smartphones by enabling flexible time manipulation with speech. We designed and developed Data@Hand, a mobile app that leverages the synergy of two complementary modalities: speech and touch. Through an exploratory study with 13 long-term Fitbit users, we examined how multimodal interaction helps participants explore their own health data. Participants successfully adopted multimodal interaction (i.e., speech and touch) for convenient and fluid data exploration. Based on the quantitative and qualitative findings, we discuss design implications and opportunities with multimodal interaction for better supporting visual data exploration on mobile devices.
more »
« less
- Award ID(s):
- 1753452
- PAR ID:
- 10315476
- Date Published:
- Journal Name:
- 2021 CHI Conference on Human Factors in Computing Systems
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Eating disorders (EDs) constitute a mental illness with the highest mortality. Today, mobile health apps provide promising means to ED patients for managing their condition. Apps enable users to monitor their eating habits, thoughts, and feelings, and offer analytic insights for behavior change. However, not only have scholars critiqued the clinical validity of these apps, their underlying design principles are not well understood. Through a review of 34 ED apps, we uncovered 11 different data types ED apps collect, and 9 strategies they employ to support collection and reflection. Drawing upon personal health informatics and visualization frameworks, we found that most apps did not adhere to best practices on what and how data should be collected from and reflected to users, or how data-driven insights should be communicated. Our review offers suggestions for improving the design of ED apps such that they can be useful and meaningful in ED recovery.more » « less
-
more » « less
Mobile health applications and devices (“mobile health apps”) play increasingly important roles in the lives of individuals interested in self-regulating their personal health behaviors. While some appear to be simply consumer products and services, many are embedded in regulatory programs aimed at compliance with expert guidelines. In this paper, we draw on de Vaujany et al.’s framework for organizational IT-based regulation systems to consider how systems operate in open and distributed contexts in which actors have strong agency and regulation is indirect and voluntary. To do so, we consider how IT artifacts become embedded in practices, how data are implicated in regulatory feedback loops, and how individual, organizational and technological actors are mobilized and with what regulatory outcomes. We develop an instrumental case study as a vignette of five regulatory episodes (continuous glucose monitoring systems used by persons with diabetes) to examine how expert rules materialized in mobile health apps, data about bodily states, and IT features such as displays and alarms “nudge” individuals towards compliance with self-regulatory guidelines and practices. Through this analysis, we identify two related regulatory affordances of mobile health apps for predicting and surveilling personal health. We theorize how multilevel networks composed of trifecta of rules, IT artifacts, and practices develop as a regulatory lattice through which social regulation is realized. We conclude by considering the broader implications of this analytical approach to study voluntary, data-enriched regulatory systems.
-
Data physicalizations (3D printed terrain models, anatomical scans, or even abstract data) can naturally engage both the visual and haptic senses in ways that are difficult or impossible to do with traditional planar touch screens and even immersive digital displays. Yet, the rigid 3D physicalizations produced with today's most common 3D printers are fundamentally limited for data exploration and querying tasks that require dynamic input (e.g., touch sensing) and output (e.g., animation), functions that are easily handled with digital displays. We introduce a novel style of hybrid virtual + physical visualization designed specifically to support interactive data exploration tasks. Working toward a "best of both worlds" solution, our approach fuses immersive AR, physical 3D data printouts, and touch sensing through the physicalization. We demonstrate that this solution can support three of the most common spatial data querying interactions used in scientific visualization (streamline seeding, dynamic cutting places, and world-in-miniature visualization). Finally, we present quantitative performance data and describe a first application to exploratory visualization of an actively studied supercomputer climate simulation data with feedback from domain scientists.more » « less
-
Background: Personal health technologies, including wearable tracking devices and mobile apps, have great potential to equip the general population with the ability to monitor and manage their health. However, being designed for sighted people, much of their functionality is largely inaccessible to the blind and low-vision (BLV) population, threatening the equitable access to personal health data (PHD) and health care services. Objective: This study aims to understand why and how BLV people collect and use their PHD and the obstacles they face in doing so. Such knowledge can inform accessibility researchers and technology companies of the unique self-tracking needs and accessibility challenges that BLV people experience. Methods: We conducted a web-based and phone survey with 156 BLV people. We reported on quantitative and qualitative findings regarding their PHD tracking practices, needs, accessibility barriers, and work-arounds. Results: BLV respondents had strong desires and needs to track PHD, and many of them were already tracking their data despite many hurdles. Popular tracking items (ie, exercise, weight, sleep, and food) and the reasons for tracking were similar to those of sighted people. BLV people, however, face many accessibility challenges throughout all phases of self-tracking, from identifying tracking tools to reviewing data. The main barriers our respondents experienced included suboptimal tracking experiences and insufficient benefits against the extended burden for BLV people. Conclusions: We reported the findings that contribute to an in-depth understanding of BLV people’s motivations for PHD tracking, tracking practices, challenges, and work-arounds. Our findings suggest that various accessibility challenges hinder BLV individuals from effectively gaining the benefits of self-tracking technologies. On the basis of the findings, we discussed design opportunities and research areas to focus on making PHD tracking technologies accessible for all, including BLV people.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3411764.3445421
