Tasks or procedures are inherently made up of distinct, sequentially linked steps, and we undertake many such tasks daily, from cooking to using machinery. These tasks are often complex, leading to numerous questions, particularly for those who are untrained or first-time users. Consulting a manual or instruction sheet can help, but research indicates that people often struggle with instruction manuals, thereby attempting tasks without resolving their queries, leading to errors [60]. Such errors can be crucial in many cases, such as using COVID-19 test kits, and research shows that many participants (close to 20% in one study) made critical errors while using these test kits [59]. Consequently, HCI researchers have explored computational methods to address users' queries during task execution to prevent mistakes and enhance user autonomy. Among these, dialogue-based question-answering (referred to as Q&A in the rest of the paper) systems are increasingly popular, where users What should I do next with this?
You can wash the portafilter with water.
Task knowledge (e.g., machine manual) PrISM-Tracker HAR Fig. 1. PrISM-Q&A is a novel interaction for question answering (Q&A) for supporting procedural tasks, namely, smartwatchbased step-aware Q&A. The system estimates the user context through procedure tracking based on Human Activity Recognition (HAR) and uses the information to augment the Large Language Model (LLM)'s context prior to response. For example, if a user does not know the tool's name and asks a question using a pronoun, the system can resolve the ambiguity based on the step information estimated from the smartwatch's sensors without using a camera.
can pose questions and receive answers via speech from hands-free voice assistants without interrupting their primary task [18,30]. Recently, the advancement of Large Language Models (LLMs) [77,81] has significantly improved natural language processing to achieve accurate Q&A systems.
However, even voice assistants powered by such models are often inadequate as they rely solely on the information verbalized by the user. Various studies have pointed out that articulating questions to get desired answers can be challenging, and users suffer from nonsensible responses due to assistants' misunderstanding [4,27,76]. The lack of context has been identified as the major challenge in supporting users in complex tasks with voice assistants [30,66]. Lin et al. [42] studied users' interactions with voice assistants while they cooked and found users often assumed that the assistant had a contextual understanding and asked questions such as "What is the next step?" and "How long does it last?", which are inherently hard to answer by common voice assistants.
In this paper, we introduce PrISM-Q&A, a context-sharing method for dialogue-based Q&A using a smartwatch (Figure 1). The core idea lies in its step awareness, estimating user steps during procedural tasks using multimodal Human Activity Recognition (HAR), information that is then used to enrich the user's verbal questions, helping to clarify their intent and ultimately provide more contextually appropriate responses from a speech-based assistant. For instance, if the system detects that the user has just dumped coffee grounds from a filter when asked, "What should I do with this?", the system can specifically advise, "You can wash the portafilter with water, " by resolving the ambiguity in the original question with the estimated step information and retrieving reference from a task-specific knowledge source. This step-aware Q&A interaction can enhance the answer quality while allowing the user to remain somewhat vague in their queries by integrating the user's sensed state in a task while prompting LLMs, thus offering a more intuitive Q&A experience.
Our initial study assessed how users perceived the experience of our proposed step-aware Q&A interaction using a smartwatch. We contrasted our system with existing voice assistants with the existing voice assistants (1) using only what the user asked (like Amazon Alexa) and (2) also using visual information (like visual Q&A models such as Gemini [17] and GPT-4V [54]). Using the Wizard-of-Oz methodology [12], participants (𝑁 = 8) performed one of two procedural tasks -cooking or latte-making -and responded to survey questions afterward. The study revealed that the step-aware Q&A interaction was preferred for two reasons: the intuitiveness of not having to describe detailed information about the current situation, and the physical comfort of not having to wear a camera.
We then implemented PrISM-Q&A by introducing a step-aware generator with LLMs and a step-aware retriever in conjunction with Retrieval Augmented Generation (RAG) [19] to utilize external knowledge sources, such as machine manuals. We evaluated its performance in three tasks: cooking, latte-making, and skin care. We compared PrISM-Q&A with an LLM-based Q&A pipeline that did not utilize step context and demonstrated the superiority of our step-aware approach, confirmed with multiple state-of-the-art LLMs used as a generator. In the best-performing configuration, when rated on a scale of 1 to 5 (best), the answer quality improved from 2.9 to 4.5, from 3.4 to 4.4, and from 3.9 to 4.5, for the cooking, latte-making, and skin care tasks, respectively. An ablation study suggested the effectiveness of step-aware query translation in the retriever, which increased the contextual precision in the retrieved reference. Moreover, our qualitative analysis illustrated that step information estimated by HAR helped LLMs answer questions related to procedures (e.g., "What is the next step?") and resolve ambiguities in user queries (e.g., "Where should I place this?").
Finally, we developed a real-time system using a consumer smartwatch and conducted a user study with the prototype (𝑁 = 10) in the latte-making task. This study demonstrated that the system helped novice users conduct the task by accurately responding to their questions, which are often ambiguous to answer without the context provided by the activity recognition. Moreover, the participants' post-hoc comments shed light on important areas to improve the system from the human-AI interaction perspective [1], such as increasing transparency and implementing efficient error recovery. Based on the findings, we implemented an enhanced version of the prototype and discussed future directions to further explore context-aware task assistants.
This paper makes the following contributions:
(1) A novel smartwatch-based Q&A interaction for procedural tasks, step-aware Q&A, which offers a preferable user experience to conventional voice assistants by sharing step context between the user and the system. (2) An approach to achieving the step-aware Q&A that integrates the outputs of multimodal procedure tracking into LLMs by augmenting the user's question with step context. (3) A comprehensive study across multiple task datasets to verify the proposed step-aware generator's and retriever's effectiveness in increasing factual correctness and overall quality in answers. (4) A real-time smartwatch system, whose effectiveness in supporting novice users was confirmed through a user study.
We make the implementation and datasets publicly available to support further research in this domain at https://github.com/cmusmashlab/prism. Our dataset will enable researchers to investigate the emerging field of integrating non-visual multimodal sensors with LLMs in an effort to ground AI in the physical world [73].
We first review the supporting systems for procedural tasks and highlight the need for Q&A systems that answer various user questions. Then, we discuss such dialogue systems, focusing on how the system and user share context to achieve natural and accurate Q&A. Finally, we review existing techniques for Q&A and discuss their potential to support procedural tasks.
Many HCI systems have been proposed to support users in performing various procedural tasks, including daily activities [23,26,52,67]. For example, Hamada et al. [23] introduced Cooking Navi, an interface providing multimedia recipe information to aid in cooking. HoloAssist [67] is a system where a human observer watches the task performer's first-person view captured by Augmented Reality (AR) glasses and guides them through verbal instructions. As illustrated by these examples, users often seek information while performing tasks because understanding the task completely by watching instruction videos or manuals is often challenging [42,61,79].
Here, voice assistants such as Google Home, Amazon Echo, and Apple Siri are popular methods to provide information so that users do not have to stop the primary task while maintaining agency [27] (e.g., cooking [29], manufacturing [68], medical procedure [13]). Diederich et al. [14] conducted an intensive review of research on conversational agents and highlighted the under-explored yet promising role of voice assistants for physical tasks. Qualitative insights were shared by Vtyurina and Fourney [66], who analyzed user questions during a cooking task using the Wizard-of-Oz method and reported the largest class (over 30%) of questions were inquiries about the next steps. In this work, we aim to develop a real-time system designed for procedural tasks that supports the user and answers their queries using the context provided by a common device -a smartwatch.
The vision of ambient voice assistants has been studied in HCI for a long time and is one of the important subfields of Ubicomp. Multiple studies have concluded that sharing context is crucial in developing dialogue-based humancomputer interaction [7,11,65], since users often struggle to verbalize queries well enough to get the desired responses [4,76] and want to rely on indefinite reference like pronouns [21]. For instance, Völkel et al. [65] investigated the imaginary "perfect" interaction for voice assistants and suggested that knowledge about the user and the world makes dialogues more effective and natural by creating the impression of shared knowledge and common ground.
In this regard, recent advancements in computer vision and natural language processing now allow dialogue systems to incorporate visual context more easily, with models like Gemini [17] and GPT-4V [54] leading the way. The HCI community has shown that such interactions enhance the naturalness of posing queries about the physical world [22,24,40,46,47]. For instance, GazePointAR [40] is a context-aware voice assistant designed for AR devices. This system utilizes eye gaze and pointing gestures, enabling LLMs to clarify speech queries such as "What is this?", realizing more intuitive Q&A experiences. Such a vision of smart assistants in the physical world has been popularly explored these days, leading to emerging wearable products Ray-Ban Meta Glasses [48]. Therefore, applying these advancements in visual Q&A to procedural task interactions appears promising.
However, the richness of the context shared by cameras also raises concerns about privacy, power consumption, accessibility, and the inconvenience associated with using specific devices that require a camera to maintain a clear view of user actions. The vision-based approaches also often suffer from issues related to the limited field of view, occlusion, and motion blur [51]. Some of these issues around using cameras were highlighted in recent press coverage of now-closed Amazon Go Stores, where a seemingly easy task of monitoring customers required several cameras to get a good view of the users and their actions as well as an army of crowd-workers to label and verify inferences. Moreover, it was perceived as a privacy nightmare 1 .
Still, prior research showed that users frequently seek procedural information such as "What is the next step?", "Anything else I haven't done?", and "How long does it last?" [42,66]. This insight and the challenges of using a camera guided our hypothesis: It is possible to compensate for the lack of shared visual context with a voice assistant by providing context related to the user's status within the procedure.
This hypothesis gained further support from various sensing approaches explored in the Ubicomp community, specifically Human Activity Recognition (HAR). For example, smartwatch-based HAR has been widely studied using audio and motion sensors [5,9,20,[37][38][39]50]. The potential to track user steps using these sensors has been confirmed in various procedural tasks, such as cooking a brownie [63] and fried noodle [36], insulin self-injection [8], and assembly work [72]. Our prior work, PrISM-Tracker [3] utilized transition information of procedural tasks for enhanced procedure tracking in latte-making and wound care tasks. This work aims to integrate such multimodal HAR into Q&A systems for procedural tasks as a novel context-sharing approach.
Recent advancements in Large Language Models (LLMs) have significantly impacted the field of natural language processing, particularly in question-answering (Q&A) tasks [77,81]. In several studies, LLMs have demonstrated impressive performance in general open-domain Q&A tasks (e.g., Wikipedia) and data-driven response generation [15]. However, research has also shown that LLMs struggle to learn long-tail knowledge [34,44], resulting in errors [33] that degrade the system performance and fail to meet user expectations. In response, Retrieval Augmented Generation (RAG) is a technique in achieving more reliable Q&A systems [19,41]. Using RAG, the system refers to a knowledge source in response to a user question, finds relevant information, and generates an answer based on the reference, often using LLMs. Studies have shown that RAG-augmented models can outperform traditional LLMs, especially in domains where precision and factual correctness are paramount [41]. Procedural tasks are typically accompanied by knowledge sources, such as recipes for cooking, instruction sheets for medical kits, and manuals for machinery. Therefore, LLM-based Q&A systems referencing such information when generating answers hold promise. Our work seeks to enhance this approach by integrating sensed user context in the physical world.
As outlined in Section 2.2, providing contextual information to Q&A systems is critical, yet current methods mainly rely on users' verbal descriptions or shared visual data. To develop supporting systems for procedural tasks, we introduce a step-aware Q&A, where the system uses sensor data to develop an understanding of which step of a procedure the user is performing. This awareness about the step is combined with the fixed description of the procedure itself before it is provided as the context to the language model. The goal of our work is to evaluate the utility of the sensed information as part of the shared context.
Thus, we first aim to understand how the shared context provided by the sensed information influences user interaction with the system: RQ1: How do users use and perceive smartwatch-based step-aware Q&A compared to conventional voice assistants? Second, assessing the technical feasibility of step-aware Q&A using a smartwatch is essential. We chose smartwatches for their ubiquity and minimal privacy issues relative to camera-based systems, as well as their ability to monitor various daily activities. Considering recent LLMs' capability in Q&A and HAR studies using a smartwatch, as discussed in Section 2, we consider: RQ2: Can LLM-Based Q&A pipelines effectively use context estimated from smartwatch's sensor data to achieve step-aware Q&A?
As we aim to assist users in performing procedural tasks more effectively, it is necessary to test the user experience with a real-time prototype, leading to our final research question: RQ3: Is the real-time step-aware Q&A system helpful in supporting users' needs during procedural tasks?
We first explored how users perceived our proposed step-aware Q&A interaction to address RQ1. We employed the Wizard-of-Oz methodology [12], commonly used in dialogue system evaluation, to simulate this new interaction and assess user perceptions.
This study utilized a within-participant design, comparing three conditions: voice-only, vision-based, and sensorbased. The voice-only condition served as a baseline, where the Q&A system only used information verbally provided by the user. The vision-based condition incorporated visual data into the Q&A process, an interaction becoming common with emerging smart glasses such as Ray-Ban Meta Glasses [48]. The sensor-based condition, our focal condition, leveraged HAR techniques to inform the Q&A system of the user's step within a procedural task using a smartwatch. We assumed that both approaches could track user steps perfectly, which was an unrealistic setting, but for this experiment, we focused on exploring user perception in ideal dialogue scenarios. A more detailed evaluation of response accuracy with actual sensor data will follow in Study 2. The wizard responded to participants' questions according to each condition, which is described in Appendix C.1.
We used two tasks: cooking and latte-making. The cooking task involved preparing a sunny-side up egg and grilling a sausage, with participants likely asking a variety of questions related to the recipe and cookware use. The latte-making task required using a machine to make a latte, potentially prompting questions about more complex operations detailed in the manual. Each task's detailed procedural steps are outlined in Appendix A. Participants were allowed to modify the order of steps freely.
We recruited eight participants (P1-P8, 6 male, 2 female; aged Mean = 39.4, SD = 18.2) via word-of-mouth from our institution and the local community. Half of the participants (P1-P4) were assigned to the cooking task, while the other half undertook the latte-making task (P5-P8). We assessed their familiarity with these tasks by asking how frequently they performed the task, which we used as a measure of their proficiency. They got the cooked meal or cup of latte they made in the session as compensation for their participation.
We assessed usability using the System Usability Scale (SUS) [10]. This technology-independent measure is widely used for subjective evaluations of system usability. The SUS consists of 10 items, and participants respond to each item using a 5-point Likert scale where 1 represents 'strongly disagree,' and 5 represents 'strongly agree.' Scores are then calculated on a scale from 0 to 100.
Each participant was initially briefed on the assigned task and the three Q&A conditions they would encounter during the task execution. The task was segmented into three equal parts, each of which was assigned a Q&A condition, with the order in which conditions were experienced randomized. At the start of each segment, we paused the task to inform participants of the upcoming condition. For the vision-based condition, participants wore mock AR glasses as part of a simulated setup, while for the voice-only and sensor-based conditions, they wore a smartwatch. Given the Wizard-of-Oz methodology, these devices were non-functional; however, participants were instructed to imagine these were operational and to internalize their experiences to prepare for a posttask questionnaire. They were also encouraged to ask at least three questions in each segment to facilitate the internalization. Following the task, participants completed the questionnaire, providing ratings for SUS and open-ended feedback about their overall experience, preferred conditions, and the reasons for their preferences.
4.6.1 Question Types. We noted a pattern in questions participants posed under each experimental condition. In the voice-only and sensor-based conditions, questions primarily sought factual information likely available in a manual. For instance, during a latte-making task in the voice-only condition, a participant asked, "I'm grinding beans. How long should I wait for one-shot beans to be ground?" [P6, experienced user]. Similarly, in the sensor-based condition during a cooking task, the question was, "How long should I wait? [while grilling a sausage]" [P2, experienced user]. Conversely, in the vision-based condition, participants asked not only factual questions but also those requiring visual information to answer, such as "Is this the right angle to hold the milk jar?" [P8, experienced user]. Notably, in all conditions, the participants often asked about the next step, which is aligned with the findings by Vtyurina and Fourney [66].
Additionally, there was a notable shift in language use. In the voice-only condition, participants provided more detailed information to clarify, like, "I have finished grinding beans. What is the next step?" [P8]. This tendency reflects users' efforts to increase the likelihood of receiving the desired response [43,57]. On the other hand, in the vision-based and sensor-based conditions, they often used pronouns to refer to objects or actions or omitted the referent, for example, "What should I do next?" [P7, novice user] or "Where should I attach this?" [P7]. Such change is understood given that users desire to communicate to voice assistants using pronouns [21].
From these observations, we hypothesized that although questions demanding visual or world information are still challenging, the sensor-based Q&A could facilitate inquiries about factual knowledge by contextualizing the users' state within the procedure. Specifically, we anticipated that the capability might allow it to resolve ambiguous queries like "What should I do next?" or "Where should I attach this?" based on the step context, thus reducing the need for users to elaborate extensively on their situations when posing questions.
4.6.2 User Experience. The SUS scores for the vision-based (Mean = 76.3, SD = 13.3) and sensor-based (Mean = 81.0, SD = 12.1) conditions were higher than the voice-only (Mean = 63.1, SD = 13.4). This improvement was statistically significant, as confirmed by a 𝑡-test with Bonferroni correction following a one-way ANOVA. However, there was no significant difference between the vision-based and sensor-based conditions (𝑝 > 0.05). By looking at their comments, we found the participants appreciated the context-sharing aspects of the interactions, stating benefits such as, "Using the Q&A-based task requires a strong learning curve since you have to describe the questions in more detail"[P8, voice-only]foot_6 , "It was helpful to ask questions about the instruction of the task like what to do next." [P2, sensor-based], and "It seems very easy to use this kind of system as it is powerful enough to recognize the objects that I am facing and understand my intent. " [P6, vision-based].
Moreover, the different modalities and devices used in the two conditions led to varied user experiences. Some participants preferred the convenience of a smartwatch over AR glasses, noting, "While the glasses would be powerful, I preferred using the smartwatch over AR glasses. Covering my vision is cumbersome" [P3, experienced user] 3 . Another commented on specific pros and cons: "From the accuracy of assistance point of view, I would prefer the AR. But putting on a headgear might be a little troublesome compared to Siri." [P7]. These comments highlighted the trade-offs between device convenience and the types of shared context.
Some participants also elaborated on potential use cases for each context-sharing Q&A interaction. For instance, "If I want to ask about objects around me, the vision-based Q&A is helpful. At the same time, the sensor-based Q&A seems convenient for questions about the task flow, like asking for instructions based on what I am doing. " [P1, novice user]. This comment suggested that, while both vision-based and sensor-based interactions could allow questions like "What is the next step?", the smartwatch-based system was preferred for convenience, provided that it could track user steps accurately. Another added, "The system knowing what I am doing is helpful for me to check what I need to do to finish the task properly. Sometimes, I want to refer to visual information, like to confirm the way I hold the milk jug is correct. " [P5, novice user]. Moreover, one participant suggested a hybrid approach, "I may rely on the watch first and, if needed, I'd use the camera approach like smartphone. " [P4, experienced user].
As an answer to RQ1, we concluded that the sensor-based Q&A, under the condition of ideal sensing capability, offers a preferable experience for its convenience. Specifically, the asked question types and user comments highlighted its potential to allow users to seek factual information and next steps without fully describing their current status. While incapable of asking questions that require visual information, the handiness of using a smartwatch alone appeared promising to users. Simultaneously, this novel interaction setting (i.e., knowing steps but no access to visual information) affected the language users employed (e.g., pronoun, omission), which would influence the system's Q&A performance. This insight led us directly to RQ2 concerning the system's ability to accurately respond to user questions posed under the sensor-based condition.
This section presents our approach to achieving the step-aware Q&A by connecting multimodal HAR and LLMs.
The overview of the proposed approach is shown in Figure 2. First, the sensor data from the smartwatch is processed with PrISM-tracker, introduced in our prior work [3]. This module takes in audio and motion data as well as a graph structure summarizing the transition information, i.e., how long a user is supposed to spend at each step and the transition probability between steps. Sample transition graphs are presented in Figure 6 in Appendix A. The tracker outputs frame-level step prediction, which corresponds with the window size of 0.2 seconds. This information is passed along with the user question to two components of our Q&A pipeline: step-aware generator and step-aware retriever. The retriever is only used when the system needs to refer to an external knowledge source (e.g., machine manual) to fetch references as a RAG mechanism. The generator is an LLM to synthesize the final response. We describe each component in the following subsections.
A retriever is a common technique used to provide LLMs with reference information based on a vector search using the query embedding and the knowledge source. Query translation is widely explored to rephrase the user query such that their embedding becomes more plausible and diverse to get better references [45]. In this You are an assistant for question answering to support users doing [task name] task. Use the following pieces to answer user questions. If you don't know the answer, just say that you don't know. Use one sentence maximum and keep the answer concise. User task : [task description] Users may not follow the step sequence strictly. They ask questions at some point while performing the procedure. When you refer to a step in your answer, don't mention the step number, as users do not know it. Instead, describe the step content.
As a reference, information on what the user is doing when they ask the question, which is estimated from audio and motion data on their smartwatch, is provided. When you answer the question, imagine the user performing the task according to this information and consider the context of the question. Note that this estimation is not always perfect, and you should not rely on it too much when answering the question.
The user is [probability nuance] at [current step]. If the user is at [current step], the next step is [next step]. Think step by step. Question: [user question] Context: [context] Answer: regard, we observed in Study 1 that the participants tended to omit the description of what they were doing in the step-aware interaction. Often, such context is important to identify the relevant document in the knowledge source; for instance, machine manuals are often organized roughly by steps. Therefore, we introduced a query translation based on the HAR results. Specifically, the approach augmented the user question with the estimated step information. For example, if the original question is "What should I do next?" and the system detects the user is washing hands, the translated question is "What should I do next? I'm washing my hands. " Notably, we did not assume a particular structure in the knowledge source. If, for instance, the knowledge source has a step-by-step structure that matches with the HAR classes, the retriever could pull up the corresponding documents more accurately than the embedding-based search. Yet, there are many cases where such a structured knowledge source is not available, and PrISM-Q&A was developed to generalize to such scenarios. The translated question was used in the RAG mechanism, for which detailed information is provided in Appendix B.1.
A generator is an LLM that synthesizes the final response to the user based on a given prompt. Figure 3 presents the prompt used in our step-aware generator. The [task name] and [task description] are predetermined, and [context] comes from the output of the retriever. Note that if the pipeline does not involve the retrieve, the line with [context] is removed. The [current step] comes from PrISM-Tracker, which is the step having the highest likelihood. The [probability nuance] is based on the confidence level 𝑝 of the current step, say, washing hands, it would be 'most likely washing hands' (𝑝 ≥ 0.85), 'likely washing hands' (0.85 > 𝑝 ≥ 0.7), or 'maybe washing hands' (otherwise). This threshold was determined arbitrarily. Moreover, based on the estimated current step, transition history, and the task's transition graph, it is possible to infer the possible next steps. We provided such information explicitly in LLM's prompt ([next step]) as asking about the next steps is a common interaction during procedural tasks, as suggested by prior work [66] and reaffirmed in Study 1.
We developed a real-time system using an Apple Watch (Series 7), as shown in Figure 2. The system used a laptop (MacBook Pro with 16GB Apple M1 Chip) as the computation server, and we left the self-contained system as future work. In this prototype, the smartwatch plays the role of streaming audio and motion data to the laptop wirelessly, where PrISM-Tracker and the step-aware Q&A pipeline run. The tracker is continuously applied unless a user asks a question. Users can initiate a question with the wake-up word "Hey PrISM, " a defacto approach in voice assistants. When a speech is detected, the HAR is stopped so as not to add noise to the tracker, which means the tracker uses the latest tracking output before the user mentions the wake-up keyword. The user question is transcribed using OpenAI Whisper API [53]. The question and the estimated contextual information are then fed into the step-aware Q&A pipeline. The generated answer is converted into audio using OpenAI Text-to-Speech API and played on the laptop, during which the tracker is also stopped.
We did a speed test for each module. The average process time for our step-aware Q&A was 2.8 seconds with RAG and 1.7 seconds without RAG when GPT-4-turbo API was used as an LLM. Additional 1.7 seconds and 1.8 seconds were added for the Whisper and Text-to-Speech API, respectively. Note that the PrISM-Tracker was continuously applied to each frame of 0.2 seconds in parallel. This process was lightweight, taking 0.04 seconds on the laptop.
In Study 2, we evaluated the Q&A performance of the proposed approach to investigate RQ2. To do this, we first created procedure Q&A datasets in multiple daily tasks where multimodal sensor data from a smartwatch, questions, and their timing were curated. Then, we compared our proposed Q&A approach with baselines.
To test the performance of the proposed step-aware Q&A method, we needed a dataset with questions relevant to specific moments of procedural tasks. Here, to rigorously examine model capability, it is important to collect a large pool of questions users might ask spontaneously. Having participants ask many questions while performing tasks was costly and cognitively highly demanding. Thus, we took an offline question synthesis approach with prerecorded session data; we collected sensor data of users doing the task, and a different set of participants watched task videos later and generated many potential questions as if they were doing the task themselves. This approach, however, might result in a different set of questions from the ones that would occur in actual interactions, and thus, we will evaluate our real-time system's efficacy with novice users later in Study 3.
To explore the performance in various scenarios, we used three procedural tasks: cooking, latte-making, and facial skin care. Cooking is a popular scenario for voice assistant research [30,66], while latte-making is a complicated machine-use task that involves a long manual document to read. Facial skin care was added to investigate the approach outside the kitchen context and in a situation where wearable cameras (e.g., AR glasses) are not suitable. We used the multimodal sensor dataset introduced by Arakawa et al. [3]. The latte-making dataset consisted of 22 sessions done by 15 participants who wore an Apple Watch collecting motion and audio data, including multiple steps of using a latte-maker machine, which has a 28-page long manual book 4 . The cooking dataset consisted of 17 sessions with 8 participants, in which people cooked a sunny-side up and grilled sausage. The skin care dataset consisted of fewer sessions, 5 sessions with 5 participants, in which people washed their faces and did a moisturizing routine. Note that the participants flexibly decided the order of steps to perform in these procedural tasks, and their familiarity with the tasks differed, resulting in varied behavior in terms of time and transition patterns, as shown in Figure 6 Appendix A. These three tasks encompass a range of procedural complexity, as presented by the different number of branching paths within each task.
The datasets had the video-recorded data synchronized with the watch's sensor data with the participants' permission and appropriate ethics board approval. We used these videos to collect questions that might be asked during the tasks. For each task, we recruited ten volunteers from our institution and the local community through word-of-mouth, allowing participants to overlap across tasks. All participants who made questions for the cooking task had a basic understanding of cooking. Eight of the participants who created questions for the latte-making task had never used the machine before, while two were frequent users. None of the participants who created questions for the skin care task routinely performed it. We had them watch a randomly chosen video of each task individually and come up with potential questions. For this, we developed the simple annotation interface shown in Figure 4. Within the tool, volunteers played and stopped the video at will and posted questions as if they were doing the task. This tool collected their video identifier, posed questions, and corresponding timestamps, later used to synchronize the questions with the watch's sensor data.
Before using the tool, we explained the concept of our step-aware Q&A -the system did not "see" what the users were doing, but it could sense users' actions based on the sensors on the watch. We encouraged participants to ask questions naturally as if they had done the task. We also mentioned that it was possible to ask questions about events that might not be in the video, such as potential error cases (e.g., "The machine is not functioning. What should I do?"). For cooking and latte-making, we asked each volunteer to generate at least five questions for each task video. For skin care, since it is shorter with fewer steps, we asked them to generate at least three questions. As a result, we obtained 54, 68, and 32 questions for the cooking, latte-making, and skin care tasks, respectively.
Two external task designers made a reference answer to each question individually. In doing so, they referred to each instruction (plus the machine manual for the latte-making task) as the knowledge source for each task to develop their answers. Moreover, if a question was related to a transition in the procedure (e.g., "What should I do next?"), and there could be multiple answers, all possible answers were regarded as the correct answer. After generating each set independently, they discussed and agreed on the final answer, which we treated as the reference answer for the later evaluation.
The three tasks used in Study 2 covered different scenarios: referring to a large external knowledge source (i.e., the machine manual) or not (i.e., the instruction), both of which represented plausible situations where users want to ask questions while performing the task. Whether or not to use the external knowledge source affects the internal process of assistants, as discussed in Section 5. Accordingly, we prepared the step-aware pipeline. For the cooking and skin care task, it was a generator only, and for the latte-making task, it was a RAG pipeline where a retriever was incorporated. The sensor data was processed with PrISM-Tracker, and its outputs were utilized within the pipeline. The frame-level tracking accuracy is presented in Figure 7 in Appendix A.
To examine how the addition of step context from HAR helps Q&A performance, we introduced a baseline pipeline that does not access sensor data. The baseline retriever used the user question without the translation, while the baseline generator did not incorporate the HAR output in its prompt. Specifically, it used the same prompt in Figure 3 by removing the entire second paragraph starting with "As a reference". In addition, we introduced a vanilla pipeline for the latte-making task where it only accessed the task instruction without the machine manual, serving as a condition to indicate how much existing LLMs can answer user questions without an external knowledge source as a zero-shot. Moreover, we tested our pipelines with multiple state-of-the-art LLMs as generators to examine the generalizability, including GPT-3.5-turbo (gpt-3.5-turbo-0125), GPT-4-turbo (gpt-4-turbo-2024-04-09), and Llama-3 (8B).
We followed a common approach to evaluate the Q&A performance when the reference answer is available [32]. The metrics that can be automatically computed include answer semantic similarity and factual correctness [16]. Factual correctness quantifies the factual overlap between the generated answer and the reference answer, taking a value of 0 -1. Answer semantic similarity assesses the semantic resemblance between the generated answer and the reference using a cross-encoder model and cosine similarity, taking a value of 0 -1. We used ragas-scorefoot_11 implementation for these two metrics. Each metric was calculated per question, and the scores were averaged for all the questions in each task.
In addition, since these automated evaluations, while correlating, do not perfectly reflect human preference [78], we asked the task designers who made the reference answers to rate the outputs with a score from 1 to 5, where 1 indicates a 'completely bad answer', and 5 indicates a 'completely good answer'. The annotators were not aware of the conditions of each output. They were provided with questions, their reference answers, and the generated responses. We took their average value as the score of human evaluation. To avoid degradation in their annotation quality due to the high workload, we had them annotate the outputs of the pipelines with the best LLM in the automated evaluation.
Step Awareness. The evaluation result in the three datasets is shown in Table 1, summarizing the averaged score of each metric. We found the GPT-4-turbo performed the best among the three LLMs. Moreover, within each LLM, the step-aware pipeline worked outperformed other pipelines in all tasks. When we looked at factual correctness using a paired 𝑡-test, there were significant differences in all but one case (𝑝 < 0.05): Llama-3 (8B) in the latte-making task. Interestingly, when using GPT-3.5-turbo and Llama-3 Table 1. Performance evaluation in the three procedural tasks (cooking, latte-making, and skin care), comparing three pipelines (vanilla, baseline, and step-aware). Scores range from 0 to 1, and the higher is better. FC and SS stand for factual correctness and semantic similarity metrics, respectively. The step-aware pipeline outperforms other pipelines in both metrics.
Task Cooking Latte-Making Skin Care LLM Pipeline FC SS FC SS FC SS GPT-3.5-turbo Vanilla --0.32 0.85 --Baseline 0.25 0.84 0.26 0.85 0.46 0.90 Step-Aware 0.49 0.89 0.48 0.89 0.50 0.90 GPT-4-turbo Vanilla --0.27 0.82 --Baseline 0.24 0.81 0.36 0.85 0.46 0.88 Step-Aware 0.52 0.89 0.57 0.89 0.62 0.92 Llama-3 (8B) Vanilla --0.24 0.83 --Baseline 0.24 0.83 0.16 0.78 0.40 0.88 Step-Aware 0.36 0.86 0.15 0.78 0.43 0.88 Table 2. Results of the human evaluation in the three tasks. Scores range from 1 ('completely bad answer') to 5 ('completely good answer'). The step-aware pipeline outperforms the baseline significantly (𝑝 < 0.05).
Task Cooking Latte-Making Skin Care Baseline 2.9 ± 1.7
3.4 ± 1.6 3.9 ± 1.3
Step-Aware 4.5 ± 0.94 4.4 ± 1.0 4.5 ± 1.1 (8B) in the latte-making task, the performance dropped in the baseline RAG pipeline compared to the vanilla (zero-shot) pipeline. This implied the difficulty in referring to external knowledge sources with less powerful LLMs, especially when the retriever is not perfectly fetching the desired knowledge source, as suggested by BehnamGhader et al. [6]. Simultaneously, the performance increased in the proposed step-aware pipeline with GPT-3.5-turbo and GPT-4-turbo, which implied the contribution of the enhanced retriever in addition to the generator.
Given this, we obtained the human annotation on the baseline and step-aware pipelines with GPT-4-turbo. As summarized in Table 2, the human evaluation scores were also significantly higher in the step-aware pipeline than in the baseline. The inter-annotator agreement of the human evaluation was 0.67, 0.43, and 0.62 for the cooking, latte-making, and skin care tasks, indicating a good agreement. The high scores suggested the outputs of the proposed approach were mostly perceived as good answers. Note that the number of tokens used in the API call of GPT-4-turbo per question was 887.7 (SD = 10.6), 2453.9 (SD = 409.3), and 769.1 (SD = 10.9) for the cooking, latte-making, and skin care task, respectively in the step-aware pipelines while it was 765.8 (SD = 9.1), 2203.9 (SD = 425.1), and 642.4 (SD = 9.5) in the baseline pipelines.
By looking at the outputs, we found that the proposed step-aware approach could resolve inherently ambiguous user questions, such as "What should I do next?" It was also effective in answering questions seeking factual knowledge; for instance, "Where should I pour milk?" was posed when the user took milk from the fridge. The step-aware pipeline outputted, "Pour the milk into a jug, ensuring it fills to just below the spout position, with enough milk to cover the steam wand seal, " while the baseline outputted, "Pour the milk directly into the espresso in the cup after swirling the jug to polish and reintegrate the texture." The step-aware pipeline successfully utilized the context information inferred from PrISM-Tracker to guide the user in preparing for the next step of Table 3. Results of the ablation study in the latte-making Q&A dataset. Scores range from 0 to 1, and the higher is better.
Retriever Factual Correctness Semantic Similarity Context Precision with query translation 0.57 0.89 0.41 without query translation 0.52 0.89 0.37 steaming milk, while the baseline provided misinformation by not understanding the user's step, which could have led to a negative consequence. Conversely, when HAR prediction was wrong, the step-aware pipeline could not resolve the ambiguity appropriately to questions like "What's next?", which happened when tracking was not reliable (e.g., Steps 7 and 8 in the cooking task, as shown in Figure 7 in Appendix A). This is an important limitation of the current approach, and we discuss ways to address it in Section 8.
Step-Aware Query Translation in the Retriever. We conducted an ablation study to examine the effect of the step-aware query translation in the retriever using the latte-making task dataset. The results are shown in Table 3. Here, we added a metric -context precision, which evaluates whether the chunks in the retrieved documents are relevant to the reference answer, which was also calculated with the ragas-score implementation. The results indicated the improvement of the retriever's performance (from 0.37 to 0.41 in context precision), which led to the enhanced answer of the generator (from 0.52 to 0.57 in factual correctness). The difference was not significant according to the paired 𝑡-test. Regarding this, we observed instances where the step awareness in the retriever did not enhance the context precision because the estimated step was wrong, thereby fetching irrelevant information and adding noise to the generator.
We demonstrated the proposed step-aware pipeline improved the Q&A performance significantly in all three datasets by resolving question ambiguity and complementing the context with HAR. Based on the high score in the human evaluation, we concluded that the step-aware Q&A using a smartwatch is feasible as an answer to RQ2. The effectiveness of the step awareness was confirmed with different state-of-the-art LLMs, which suggested the generalizability of the approach. At the same time, the failure cases were coupled with the error in HAR. This remaining imperfection in our pipeline, as well as the fact that questions used in this study might differ from those that would occur in actual situations, motivated us to explore RQ3 with a real-time voice assistant.
Finally, we evaluated the usability of the step-aware Q&A interaction through a user study to answer RQ3.
We tested our prototype in the latte-making task as an example scenario where users need to know detailed information to use the machine properly. The tracker was trained with the dataset used in Study 2 prior to the experiment, the details of which were presented in Appendix C.2.
We invited 10 participants (P1-P10, 8 male, 2 female; aged Mean = 30.9, SD = 14.0) who were unfamiliar with the task through word-of-mouth from our institution and the local community. Five of them (P3-P6, P8) did not know how to make a latte, three (P1, P9, P10) roughly knew the procedure but had never used a machine, and two (P2, P7) had used a different machine to make a latte. One of them (P3) had never used a voice assistant, seven (P1, P2, P4-P6, P8, P9) had used one a few times but not frequently, and two (P7, P10) were frequent users. All of them were right-handed. None of them had participated in either Study 1 or Study 2. They were compensated with $10 USD for their participation.
After consent, the participants read a brief list of steps and watched a video in which another person made a latte with each step labeled. Here, for safety, we explained that the group head and milk wand of the machine would get hot. We also emphasized that users could switch step orders within a reasonable range. In addition, we told them not to do more than one step simultaneously. Then, we explained the prototype system and the concept of the step-aware Q&A. We had a short practice session where the participants familiarized themselves with the system (i.e., wake-up keyword, response latency).
Then, the participant wore the watch on their right wrist 6 and started the task. During the task, the participant could freely ask questions to the system. Here, for simplicity, an experimenter pressed a button on the laptop to initiate the question phase when the user uttered the keyword "Hey, PrISM" before asking a question. Then, 7 seconds were allocated for the user to ask the question. On the rare occasion that the participant got stuck during the task due to nonsensible responses from the system or unexpected events, the experimenter would offer guidance. After the task, the participant filled out the SUS [10] questionnaire and answered open-ended questions: "How did the voice assistant help you complete the task?", "How did you perceive the reliability of the voice assistant?", "How did you perceive the latency in the response?", "Are there any particular questions and responses and your feelings you would like to share with us?", "Do you have any scenarios other than latte-making where the voice assistant can be helpful?", and "Tell us any comments about the experience. " The entire study took approximately 30 minutes.
7.4.1 Question Types. Participants asked 69 questions in total (i.e., Mean = 6.9, SD = 4.6 per session). In six cases, participants either could not complete their questions within the 7 seconds allotted or the speech recognition model failed significantly, resulting in losing the original question's meaning. These instances were excluded from further analysis. The experimenter checked the remaining questions and categorized them according to non-exclusive phenomena. A total of 29 questions involved indefinite reference or omission (e.g., "What's next?", "Should I do manual or automatic?", etc), 33 were about steps (e.g., "What should I do first?", "Did I forget any step?", etc), 24 were about factual knowledge within each step (e.g., "How much milk should I pour?", "I can't get out the ground bean from the filter", etc), and 5 were not answerable with the given context description or manual, requiring either visual or environment-specific information (e.g., "Where are beans?", etc). 7.4.2 Accuracy. The experimenter manually classified each generated answer as sensible or nonsensible. As a result, 46 questions were marked as sensible (73.0%). Out of the nonsensible 17 cases, 9 were due to the errors in HAR, 3 failed in the RAG-based Q&A even though the HAR was correct, and 5 were cases where the system did not respond "I don't know" even though the questions were not answerable using the given context and manual.
Out of the nonsensible cases, eight were related to the steps (e.g., "What's next?") and the error was made by the incorrect HAR. In all of these cases, the users re-asked the question by clarifying the intention. For example, P9 asked, "What should I do next?" and the system responded, "Next, you should throw away the towel" and P9 asked again, "What should I do after throwing away the towel?" and got the desired response. In the other nonsensible cases, the participants ignored the response and proceeded by guessing themselves. For instance, P2 wanted to check if the basket was properly attached to the portafilter and asked, "Is this attaching ok?" and the system responded, "Yes, the attachment is okay if you have selected the correct filter basket for the number of coffees or strength you are making and ensured it is properly inserted into the portafilter." The question essentially required visual information to answer, and the response was not ideal, but P2 continued the task after pushing the basket a few times. These results imply the importance of transparency in the dialogue (i.e., why the assistant said what it said), as users may not have a clear idea about the system's sensing capability. 7.4.3 User Experience. The SUS score indicated good usability (Mean = 79.6, SD = 9.1). To further understand the user experience, we analyzed their answers to the open-ended questions using thematic analysis. This involved manual coding of the responses to identify and categorize recurring themes and patterns within the data.
All participants found the Q&A function helpful, "It answered my questions that I did not come up with in the beginning but later encountered while doing it" [P2]. They appreciated the assistant resolved their step-related questions, "It helped me confirm where to start in the process, what the next step was, and whether I missed any steps. I wasn't familiar with this exact machine so the assistant helped me confirm I am following the correct process" [P7]. Five participants (P3, P5, P7, P8, P10) found the step-awareness particularly helpful, "It was more reliable than I thought as it indeed gave me the answer I was looking for without mentioning the details of what I was doing" [P5].
At the same time, three participants (P6, P7, P9) mentioned the error in the step awareness, "There was one time when the step was not correct based on the assistant's answer, and I knew it was wrong, so I skipped one step. That made me think a little bit that the assistant was not very reliable" [P6]. This comment suggested that offering transparency and error recovery is critical in developing context-aware voice assistants. This is even more remarkable given that the majority of errors (9 out of 17 failure cases) were due to imperfections in the smartwatch-based HAR, as discussed in Section 7.4.2. In their session, P9 understood the error in the step awareness and re-asked questions with less ambiguity, saying, "Sometimes it misunderstood my intention, but it was very easy to know that it was misunderstanding and how it misunderstood. So I could just ask another question to avoid the misunderstanding easily." Regarding transparency, two participants suggested ways to improve it, "I wanted to know if it knew my step correctly before I asked the question. Maybe showing their estimation on the watch's interface will help" [P2] and "For example, if you're doing something something now, next is bra bra bra" [P8]. Increasing the shared context in the dialogue is also suggested to be key by Jaber et al. [30]. In addition, there was a certain latency in the prototype mainly due to the heuristic of allocating 7 seconds for the user question as the minimal implementation, and four participants (P1, P2, P4, P6) requested faster interaction, mentioning "It seemed a bit slow while I guess it is faster than I manually read the manual" [P1]. Based on these comments, we developed an enhanced implementation in the next section.
Additionally, the participants suggested further interaction possibilities. P4 mentioned a desire to combine visual information to see if the milk amount is appropriate. This is an interesting direction, also suggested in the formative study, as a hybrid of sensor-based and vision-based systems. Specifically, the assistant would prompt users to use a camera to visually check the task quality only when necessary, based on the recognized context. On the other hand, P6 and P8 mentioned proactive dialogue from the assistant, mentioning, "I think it's good to have the feature that the assistant talks more even if the user does not ask questions" [P8]. This is also an interesting research opportunity to investigate such mixed initiative [25] in task support assistance by involving people with varied proficiency and needs.
From the high SUS score and the participants' positive comments, we concluded that the real-time system of PrISM-Q&A was helpful for novice users to conduct the latte-making task as an answer to RQ3. The results reconfirmed the benefit of step-awareness in resolving ambiguity, especially for procedure-related questions like "What should I do next?" which constituted roughly half of all questions. Simultaneously, the comments suggested (1) room for improvements in the current prototype in terms of transparency and speed and (2) possibilities for {{same prompt as in the previous step-aware generator}} If the question includes ambiguity (e.g., "this" and "next") and you use the currently estimated step to resolve it, mention the currently estimated step in your answer for transparency, for instance, "After [current step], do [next step]" to answer "What is next?". In this case, do not include the step number in your response. Question: [user question] Context: [context] Answer: future task-support interactions. In response, we present an enhanced system in Section 8 based on (1) and discuss implications based on (2) in Section 9.3. Additionally, it is important future work to examine the broader effect of the prototype, for instance, on the completion time of the task in comparison with existing voice assistants, by involving more participants.
Based on the feedback received in Study 3, we developed an enhanced prototype (See Video Figure). While user studies with this enhanced prototype have yet to be conducted, we anticipate that these improvements will further enhance the user experience. Here, we describe key implementations.
First, we implemented a real-time voice activity detection to find the end of the user utterance, instead of waiting for a fixed duration. This is a common technique in voice assistants, and our implementation detail is described in Appendix B.2. Next, by using Whisper.cppfoot_15 locally instead of the Whisper API, the time for Speech-to-Text was reduced to 0.4 seconds (from 1.7 seconds) on average. Also, we replaced Text-to-Speech API with the say command in the MacOS, which reduced the network latency. As a result, the latency from the moment a user finishes a question and the moment a user starts hearing the first token of the response is roughly 4 seconds and 3 seconds with and without the RAG mechanism, respectively, while it was about 14 seconds in the previous prototype. While future work, such as adopting faster local LLMs like Phi-3 [49] and real-time speech interaction models like OpenAI's advanced voice mode, is promising to further reduce the latency, the enhanced prototype offers a reasonable speed.
Secondly, we prompted the model to mention its estimated step context in answering questions that use the information, such as "What is the next step?" and "What should I do with this?" This design was motivated by the fact that most failure cases were caused by HAR errors. As suggested by the participants in Study 3, conveying the estimated step will increase the transparency in the dialogue. This approach allows users to trust the response and correct it if necessary. For instance, instead of "The next step is to take out milk from the fridge", the response is now "After brewing coffee, take out milk from the fridge. " Our finalized prompt is shown in Figure 5.
Our studies exploring research questions successfully demonstrated the efficacy of our proposed novel interaction, step-aware Q&A. Lastly, we discuss limitations and future work.
Firstly, the current pipeline treats all questions with the same, single pipeline, but the agent-based approach [62,75], where different processes or tools were chosen based on the type of questions, is promising, given there are some patterns in questions participants ask during procedural tasks. For instance, the ReAct [75] framework allows LLMs to selectively use reasoning and actions (e.g., referring to knowledge source) to solve various tasks. In our case, since users often ask about the next step during procedural tasks [66], designing a specific tool to answer such questions based on the task's transition graph will be promising, instead of providing the information in the generator's prompt as in our current approach. Such a tool could help reliably answer questions like "Did I miss any steps?" Additionally, as Yang et al. [74] did in Q&A within instructional videos, judging the answerability of questions given the context information could mitigate incorrect responses.
Moreover, the retriever's performance is crucial in the RAG mechanism in general, which was also confirmed in Study 2. The current retriever uses the similarity between the embeddings of the user query and the source document. While Study 2 showed efficacy, further improvement is expected by introducing structure into the knowledge source, for example, using knowledge graph [55]. For example, Zhou et al. [80] proposed an approach to constructing an open-domain hierarchical knowledge base of procedures. Combining the step-awareness of PrISM-Q&A with such approaches will help deploy assistants to various tasks.
We proposed PrISM-Q&A, a step-aware question-answering (Q&A) interaction designed for procedural tasks. It enhances the capability of existing voice assistants by enabling them to comprehend user context within a procedure, as inferred from multimodal Human Activity Recognition (HAR) using a smartwatch. Specifically, the output of HAR is integrated into an LLM-based generator and retriever to resolve ambiguity in user queries and synthesize more accurate answers. Our series of studies involving three daily tasks -cooking, latte-making, and skin care -confirmed the improved quality of the generated answers and the real-time system's effectiveness in aiding novice users. This research represents an initial effort to anchor LLMs in physical tasks using sensors and to design supportive interactions. Future work will explore the assistant's effectiveness across diverse scenarios to promote successful human-AI interactions. chunk is provided to the generator as [context] in Figure 3. We used the LangChain frameworkfoot_19 to implement the above.
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 8, No. 4, Article 180. Publication date: December 2024. PrISM-Q&A: Step-Aware Voice Assistant on a Smartwatch • 180:3
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 8, No. 4, Article 180. Publication date: December 2024.
https://www.thedailybeast.com/amazons-just-walk-out-frictionless-checkout-tech-is-a-privacy-nightmare Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 8, No. 4, Article 180. Publication date: December
PrISM-Q&A: Step-Aware Voice Assistant on a Smartwatch • 180:5
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 8, No. 4, Article 180. Publication date: December 2024. 180:6 • Arakawa et al.
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 8, No. 4, Article 180. Publication date: December 2024. PrISM-Q&A: Step-Aware Voice Assistant on a Smartwatch • 180:7
This notation indicates a response to a question about the specific condition.
This notation indicates a response to a question comparing all conditions. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 8, No.
4, Article 180. Publication date: December 2024.
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 8, No. 4, Article 180. Publication date: December 2024. PrISM-Q&A: Step-Aware Voice Assistant on a Smartwatch • 180:9
https://assets.breville.com/BES990/BES990_ANZ_IB_G22_FA_LR.pdf Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 8, No. 4, Article 180. Publication date: December 2024. PrISM-Q&A: Step-Aware Voice Assistant on a Smartwatch • 180:11
https://docs.ragas.io/en/latest/concepts/metrics/index.html Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 8, No. 4, Article 180. Publication date: December 2024. PrISM-Q&A: Step-Aware Voice Assistant on a Smartwatch • 180:13
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 8, No. 4, Article 180. Publication date: December 2024. PrISM-Q&A: Step-Aware Voice Assistant on a Smartwatch • 180:15
This setting matched with the dataset.
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 8, No. 4, Article 180. Publication date: December 2024. PrISM-Q&A: Step-Aware Voice Assistant on a Smartwatch • 180:17
https://github.com/ggerganov/whisper.cpp Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol.
8, No. 4, Article 180. Publication date: December 2024.
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 8, No. 4, Article 180. Publication date: December 2024. PrISM-Q&A: Step-Aware Voice Assistant on a Smartwatch • 180:19
https://assets.breville.com/BES990/BES990_ANZ_IB_G22_FA_LR.pdf Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 8, No. 4, Article 180. Publication date: December 2024.
https://www.langchain.com/ Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 8, No. 4, Article 180. Publication date: December 2024.