- Award ID(s):
- 1900638
- NSF-PAR ID:
- 10211968
- Date Published:
- Journal Name:
- UIST '20: The 33rd Annual ACM Symposium on User Interface Software and Technology
- Page Range / eLocation ID:
- 35 to 44
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
null (Ed.)We summarize our past five years of work on designing, building, and studying Sugilite, an interactive task learning agent that can learn new tasks and relevant associated concepts interactively from the user’s natural language instructions and demonstrations leveraging the graphical user interfaces (GUIs) of third-party mobile apps. Through its multi-modal and mixed-initiative approaches for Human- AI interaction, Sugilite made important contributions in improving the usability, applicability, generalizability, flexibility, robustness, and shareability of interactive task learning agents. Sugilite also represents a new human-AI interaction paradigm for interactive task learning, where it uses existing app GUIs as a medium for users to communicate their intents with an AI agent instead of the interfaces for users to interact with the underlying computing services. In this chapter, we describe the Sugilite system, explain the design and implementation of its key features, and show a prototype in the form of a conversational assistant on Android.more » « less
-
Li, Yang ; Hilliges, Otmar (Ed.)We summarize our past five years of work on designing, building, and studying Sugilite, an interactive task learning agent that can learn new tasks and relevant associated concepts interactively from the user’s natural language instructions and demonstrations leveraging the graphical user interfaces (GUIs) of third-party mobile apps. Through its multi-modal and mixed-initiative approaches for Human-AI interaction, Sugilite made important contributions in improving the usability, applicability, generalizability, flexibility, robustness, and shareability of interactive task learning agents. Sugilite also represents a new human-AI interaction paradigm for interactive task learning, where it uses existing app GUIs as a medium for users to communicate their intents with an AI agent instead of the interfaces for users to interact with the underlying computing services. In this chapter, we describe the Sugilite system, explain the design and implementation of its key features, and show a prototype in the form of a conversational assistant on Android.more » « less
-
Abstract Small‐to‐medium businesses are always seeking affordable ways to advertise their products and services securely. With the emergence of mobile technology, it is possible than ever to implement innovative Location‐Based Advertising (LBS) systems using smartphones that preserve the privacy of mobile users. In this paper, we present a prototype implementation of such systems by developing a distributed privacy‐preserving system, which has parts executing on smartphones as a mobile app, as well as a web‐based application hosted on the cloud. The mobile app leverages Google Maps libraries to enhance the user experience in using the app. Mobile users can use the app to commute to their daily destinations while viewing relevant ads such as job openings in their neighborhood, discounts on favorite meals, etc. We developed a client‐server privacy architecture that anonymizes the mobile user trajectories using a bounded perturbation strategy. A multi‐modal sensing approach is proposed for modeling the context switching of the developed LBS system, which we represent as a Finite State Machine model. The multi‐modal sensing approach can reduce the power consumed by mobile devices by automatically detecting sensing mode changes to avoid unnecessary sensing. The developed LBS system is organized into two parts: the business side and the user side. First, the business side allows business owners to create new ads by providing the ad details, Geo‐location, photos, and any other instructions. Second, the user side allows mobile users to navigate through the map to see ads while walking, driving, bicycling, or quietly sitting in their offices. Experimental results are presented to demonstrate the scalability and performance of the mobile side. Our experimental evaluation demonstrates that the mobile app incurs low processing overhead and consequently has a small energy footprint.
-
Kim, JH. ; Singh, M. ; Khan, J. ; Tiwary, U.S. ; Sur, M. ; Singh, D. (Ed.)Cyberattacks and malware infestation are issues that surround most operating systems (OS) these days. In smartphones, Android OS is more susceptible to malware infection. Although Android has introduced several mechanisms to avoid cyberattacks, including Google Play Protect, dynamic permissions, and sign-in control notifications, cyberattacks on Android-based phones are prevalent and continuously increasing. Most malware apps use critical permissions to access resources and data to compromise smartphone security. One of the key reasons behind this is the lack of knowledge for the usage of permissions in users. In this paper, we introduce Permission-Educator, a cloud-based service to educate users about the permissions associated with the installed apps in an Android-based smartphone. We developed an Android app as a client that allows users to categorize the installed apps on their smartphones as system or store apps. The user can learn about permissions for a specific app and identify the app as benign or malware through the interaction of the client app with the cloud service. We integrated the service with a web server that facilitates users to upload any Android application package file, i.e. apk, to extract information regarding the Android app and display it to the user.more » « less
-
null (Ed.)A major problem in task-oriented conversational agents is the lack of support for the repair of conversational breakdowns. Prior studies have shown that current repair strategies for these kinds of errors are often ineffective due to: (1) the lack of transparency about the state of the system's understanding of the user's utterance; and (2) the system's limited capabilities to understand the user's verbal attempts to repair natural language understanding errors. This paper introduces SOVITE, a new multi-modal speech plus direct manipulation interface that helps users discover, identify the causes of, and recover from conversational breakdowns using the resources of existing mobile app GUIs for grounding. SOVITE displays the system's understanding of user intents using GUI screenshots, allows users to refer to third-party apps and their GUI screens in conversations as inputs for intent disambiguation, and enables users to repair breakdowns using direct manipulation on these screenshots. The results from a remote user study with 10 users using SOVITE in 7 scenarios suggested that SOVITE's approach is usable and effective.more » « less