More Like this

1. The VoxWorld Platform for Multimodal Embodied Agents

   Krishnaswamy, Nikhil; Pickard, William; Cates, Brittany; Blanchard, Nathaniel; Pustejovsky, James (June 2022, Proceedings of the Thirteenth Language Resources and Evaluation Conference)

   We present a five-year retrospective on the development of the VoxWorld platform, first introduced as a multimodal platform for modeling motion language, that has evolved into a platform for rapidly building and deploying embodied agents with contextual and situational awareness, capable of interacting with humans in multiple modalities, and exploring their environments. In particular, we discuss the evolution from the theoretical underpinnings of the VoxML modeling language to a platform that accommodates both neural and symbolic inputs to build agents capable of multimodal interaction and hybrid reasoning. We focus on three distinct agent implementations and the functionality needed to accommodate all of them: Diana, a virtual collaborative agent; Kirby, a mobile robot; and BabyBAW, an agent who self-guides its own exploration of the world. 

   more » « less
2. Embodied Multimodal Agents to Bridge the Understanding Gap

   Krishnaswamy, Nikhil.; Alalyani, Nada (April 2021, Proceedings of the First Workshop on Bridging Human–Computer Interaction and Natural Language Processing)

   In this paper we argue that embodied multimodal agents, i.e., avatars, can play an important role in moving natural language processing toward “deep understanding.” Fully featured interactive agents, model encounters between two “people,” but a language-only agent has little environmental and situational awareness. Multimodal agents bring new opportunities for interpreting visuals, locational information, gestures, etc., which are more axes along which to communicate. We propose that multimodal agents, by facilitating an embodied form of human-computer interaction, provide additional structure that can be used to train models that move NLP systems closer to genuine “understanding” of grounded language, and we discuss ongoing studies using existing systems. 

   more » « less
3. Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations

   Li, Toby Jia-Jun; Radensky, Marissa; Jia, Justin; Singarajah, Kirielle; Mitchell, Tom M.; Myers, Brad A. (February 2020, The AAAI-20 Workshop on Intelligent Process Automation (IPA-20))

   Natural language programming is a promising approach to enable end users to instruct new tasks for intelligent agents. However, our formative study found that end users would often use unclear, ambiguous or vague concepts when naturally instructing tasks in natural language, especially when specifying conditionals. Existing systems have limited support for letting the user teach agents new concepts or explaining unclear concepts. In this paper, we describe a new multimodal domain-independent approach that combines natural language programming and programming-by-demonstration to allow users to first naturally describe tasks and associated conditions at a high level, and then collaborate with the agent to recursively resolve any ambiguities or vagueness through conversations and demonstrations. Users can also define new procedures and concepts by demonstrating and referring to contents within GUIs of existing mobile apps. We demonstrate this approach in PUMICE, an end-user programmable agent that implements this approach. A lab study with 10 users showed its usability. 

   more » « less
4. Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations

   Li, Toby Jia-Jun; Radensky, Marissa; Jia, Justin; Singarajah, Kirielle; Mitchell, Tom M.; Myers, Brad A. (February 2020, The AAAI-20 Workshop on Intelligent Process Automation (IPA-20))

   Natural language programming is a promising approach to enable end users to instruct new tasks for intelligent agents. However, our formative study found that end users would often use unclear, ambiguous or vague concepts when naturally instructing tasks in natural language, especially when specifying conditionals. Existing systems have limited support for letting the user teach agents new concepts or explaining unclear concepts. In this paper, we describe a new multimodal domain-independent approach that combines natural language programming and programming-by-demonstration to allow users to first naturally describe tasks and associated conditions at a high level, and then collaborate with the agent to recursively resolve any ambiguities or vagueness through conversations and demonstrations. Users can also define new procedures and concepts by demonstrating and referring to contents within GUIs of existing mobile apps. We demonstrate this approach in PUMICE, an end-user programmable agent that implements this approach. A lab study with 10 users showed its usability. 

   more » « less
5. Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challenges

   https://doi.org/10.1609/aaai.v39i27.35084  

   Reddy, Chandan K; Shojaee, Parshin (April 2025, Proceedings of the AAAI Conference on Artificial Intelligence)

   Scientific discovery is a complex cognitive process that has driven human knowledge and technological progress for centuries. While artificial intelligence (AI) has made significant advances in automating aspects of scientific reasoning, simulation, and experimentation, we still lack integrated AI systems capable of performing autonomous long-term scientific research and discovery. This paper examines the current state of AI for scientific discovery, highlighting recent progress in large language models and other AI techniques applied to scientific tasks. We then outline key challenges and promising research directions toward developing more comprehensive AI systems for scientific discovery, including the need for science-focused AI agents, improved benchmarks and evaluation metrics, multimodal scientific representations, and unified frameworks combining reasoning, theorem proving, and data-driven modeling. Addressing these challenges could lead to transformative AI tools to accelerate progress across disciplines towards scientific discovery. 

   more » « less