skip to main content


Title: DoThisHere: Multimodal Interaction to Improve Cross-Application Tasks on Mobile Devices
Many computing tasks, such as comparison shopping, two-factor authentication, and checking movie reviews, require using multiple apps together. On large screens, "windows, icons, menus, pointer" (WIMP) graphical user interfaces (GUIs) support easy sharing of content and context between multiple apps. So, it is straightforward to see the content from one application and write something relevant in another application, such as looking at the map around a place and typing walking instructions into an email. However, although today's smartphones also use GUIs, they have small screens and limited windowing support, making it hard to switch contexts and exchange data between apps. We introduce DoThisHere, a multimodal interaction technique that streamlines cross-app tasks and reduces the burden these tasks impose on users. Users can use voice to refer to information or app features that are off-screen and touch to specify where the relevant information should be inserted or is displayed. With DoThisHere, users can access information from or carry information to other apps with less context switching. We conducted a survey to find out what cross-app tasks people are currently performing or wish to perform on their smartphones. Among the 125 tasks that we collected from 75 participants, we found that 59 of these tasks are not well supported currently. DoThisHere is helpful in completing 95% of these unsupported tasks. A user study, where users are shown the list of supported voice commands when performing a representative sample of such tasks, suggests that DoThisHere may reduce expert users' cognitive load; the Query action, in particular, can help users reduce task completion time.  more » « less
Award ID(s):
1900638
NSF-PAR ID:
10211968
Author(s) / Creator(s):
 ; ;
Date Published:
Journal Name:
UIST '20: The 33rd Annual ACM Symposium on User Interface Software and Technology
Page Range / eLocation ID:
35 to 44
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    We summarize our past five years of work on designing, building, and studying Sugilite, an interactive task learning agent that can learn new tasks and relevant associated concepts interactively from the user’s natural language instructions and demonstrations leveraging the graphical user interfaces (GUIs) of third-party mobile apps. Through its multi-modal and mixed-initiative approaches for Human- AI interaction, Sugilite made important contributions in improving the usability, applicability, generalizability, flexibility, robustness, and shareability of interactive task learning agents. Sugilite also represents a new human-AI interaction paradigm for interactive task learning, where it uses existing app GUIs as a medium for users to communicate their intents with an AI agent instead of the interfaces for users to interact with the underlying computing services. In this chapter, we describe the Sugilite system, explain the design and implementation of its key features, and show a prototype in the form of a conversational assistant on Android. 
    more » « less
  2. Li, Yang ; Hilliges, Otmar (Ed.)
    We summarize our past five years of work on designing, building, and studying Sugilite, an interactive task learning agent that can learn new tasks and relevant associated concepts interactively from the user’s natural language instructions and demonstrations leveraging the graphical user interfaces (GUIs) of third-party mobile apps. Through its multi-modal and mixed-initiative approaches for Human-AI interaction, Sugilite made important contributions in improving the usability, applicability, generalizability, flexibility, robustness, and shareability of interactive task learning agents. Sugilite also represents a new human-AI interaction paradigm for interactive task learning, where it uses existing app GUIs as a medium for users to communicate their intents with an AI agent instead of the interfaces for users to interact with the underlying computing services. In this chapter, we describe the Sugilite system, explain the design and implementation of its key features, and show a prototype in the form of a conversational assistant on Android. 
    more » « less
  3. Abstract

    Small‐to‐medium businesses are always seeking affordable ways to advertise their products and services securely. With the emergence of mobile technology, it is possible than ever to implement innovative Location‐Based Advertising (LBS) systems using smartphones that preserve the privacy of mobile users. In this paper, we present a prototype implementation of such systems by developing a distributed privacy‐preserving system, which has parts executing on smartphones as a mobile app, as well as a web‐based application hosted on the cloud. The mobile app leverages Google Maps libraries to enhance the user experience in using the app. Mobile users can use the app to commute to their daily destinations while viewing relevant ads such as job openings in their neighborhood, discounts on favorite meals, etc. We developed a client‐server privacy architecture that anonymizes the mobile user trajectories using a bounded perturbation strategy. A multi‐modal sensing approach is proposed for modeling the context switching of the developed LBS system, which we represent as a Finite State Machine model. The multi‐modal sensing approach can reduce the power consumed by mobile devices by automatically detecting sensing mode changes to avoid unnecessary sensing. The developed LBS system is organized into two parts: the business side and the user side. First, the business side allows business owners to create new ads by providing the ad details, Geo‐location, photos, and any other instructions. Second, the user side allows mobile users to navigate through the map to see ads while walking, driving, bicycling, or quietly sitting in their offices. Experimental results are presented to demonstrate the scalability and performance of the mobile side. Our experimental evaluation demonstrates that the mobile app incurs low processing overhead and consequently has a small energy footprint.

     
    more » « less
  4. Kim, JH. ; Singh, M. ; Khan, J. ; Tiwary, U.S. ; Sur, M. ; Singh, D. (Ed.)
    Cyberattacks and malware infestation are issues that surround most operating systems (OS) these days. In smartphones, Android OS is more susceptible to malware infection. Although Android has introduced several mechanisms to avoid cyberattacks, including Google Play Protect, dynamic permissions, and sign-in control notifications, cyberattacks on Android-based phones are prevalent and continuously increasing. Most malware apps use critical permissions to access resources and data to compromise smartphone security. One of the key reasons behind this is the lack of knowledge for the usage of permissions in users. In this paper, we introduce Permission-Educator, a cloud-based service to educate users about the permissions associated with the installed apps in an Android-based smartphone. We developed an Android app as a client that allows users to categorize the installed apps on their smartphones as system or store apps. The user can learn about permissions for a specific app and identify the app as benign or malware through the interaction of the client app with the cloud service. We integrated the service with a web server that facilitates users to upload any Android application package file, i.e. apk, to extract information regarding the Android app and display it to the user. 
    more » « less
  5. null (Ed.)
    A major problem in task-oriented conversational agents is the lack of support for the repair of conversational breakdowns. Prior studies have shown that current repair strategies for these kinds of errors are often ineffective due to: (1) the lack of transparency about the state of the system's understanding of the user's utterance; and (2) the system's limited capabilities to understand the user's verbal attempts to repair natural language understanding errors. This paper introduces SOVITE, a new multi-modal speech plus direct manipulation interface that helps users discover, identify the causes of, and recover from conversational breakdowns using the resources of existing mobile app GUIs for grounding. SOVITE displays the system's understanding of user intents using GUI screenshots, allows users to refer to third-party apps and their GUI screens in conversations as inputs for intent disambiguation, and enables users to repair breakdowns using direct manipulation on these screenshots. The results from a remote user study with 10 users using SOVITE in 7 scenarios suggested that SOVITE's approach is usable and effective. 
    more » « less