null
(Ed.)
We summarize our past five years of work on designing, building, and
studying Sugilite, an interactive task learning agent that can learn new tasks and relevant
associated concepts interactively from the user’s natural language instructions
and demonstrations leveraging the graphical user interfaces (GUIs) of third-party
mobile apps. Through its multi-modal and mixed-initiative approaches for Human-
AI interaction, Sugilite made important contributions in improving the usability,
applicability, generalizability, flexibility, robustness, and shareability of interactive
task learning agents. Sugilite also represents a new human-AI interaction paradigm
for interactive task learning, where it uses existing app GUIs as a medium for users
to communicate their intents with an AI agent instead of the interfaces for users
to interact with the underlying computing services. In this chapter, we describe the
Sugilite system, explain the design and implementation of its key features, and
show a prototype in the form of a conversational assistant on Android.
more »
« less