Understanding what sequence of steps are needed to complete a goal can help artificial intelligence systems reason about human activities. Past work in NLP has examined the task of goal-step inference for text. We introduce the visual analogue. We propose the Visual Goal-Step Inference (VGSI) task, where a model is given a textual goal and must choose which of four images represents a plausible step towards that goal. With a new dataset harvested from wikiHow consisting of 772,277 images representing human actions, we show that our task is challenging for state-of-the-art multimodal models. Moreover, the multimodal representation learned from our data can be effectively transferred to other datasets like HowTo100m, increasing the VGSI accuracy by 15 - 20%. Our task will facilitate multimodal reasoning about procedural events.
The rational selection of goal operations and the integration of search strategies with goal-driven marine autonomy
Intelligent physical systems as embodied cognitive systems must perform high-level reasoning while concurrently managing an underlying control architecture. The link between cognition and control must manage the problem of converting continuous values from the real world to symbolic representations (and back). To generate effective behaviors, reasoning must include a capacity to replan, acquire and update new information, detect and respond to anomalies, and perform various operations on system goals. But, these processes are not independent and need further exploration. This paper examines an agent’s choices when multiple goal operations co-occur and interact, and it establishes a method of choosing between them. We demonstrate the benefits and discuss the trade offs involved with this and show positive results in a dynamic marine search task.
- Award ID(s):
- 1849131
- Publication Date:
- NSF-PAR ID:
- 10352601
- Journal Name:
- Proceedings of the Ninth Annual Conference on Advances in Cognitive Systems
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Process safety is at the heart of operation of many chemical processing companies. However, the Chemical Safety Board (CSB) has still documented over 800 investigations of process safety failures since the year 2000. While not all of these incidents were severe, some did lead to employee injuries or death and environmental harm. As a result, chemical engineering companies are increasingly dedicated to process safety through training programs and detailed vigilance as part of their operations practice. AIChE and OSHA also offer courses in process safety to help support the industry. These efforts illustrate the paramount importance that chemical engineering graduates have an appreciation and understanding of process safety as they transition from their degree program into industrial positions. Previous studies have shown that despite difficulties due to course load constraints, process safety has been incorporated into chemical engineering curriculum through either the addition of new courses, incorporation of the content within existing classes, or a combination of the two methods. A review performed in Process Safety Progress suggested that a key step for departments moving forward is to perform an assessment of the process safety culture within their institution in order to determine how faculty and students view process safety.more »
-
Process safety is at the heart of operation of many chemical processing companies. However, the Chemical Safety Board (CSB) has still documented over 800 investigations of process safety failures since the year 2000. While not all of these incidents were severe, some did lead to employee injuries or death and environmental harm. As a result, chemical engineering companies are increasingly dedicated to process safety through training programs and detailed vigilance as part of their operations practice. AIChE and OSHA also offer courses in process safety to help support the industry. These efforts illustrate the paramount importance that chemical engineering graduates have an appreciation and understanding of process safety as they transition from their degree program into industrial positions. Previous studies have shown that despite difficulties due to course load constraints, process safety has been incorporated into chemical engineering curriculum through either the addition of new courses, incorporation of the content within existing classes, or a combination of the two methods. A review performed in Process Safety Progress suggested that a key step for departments moving forward is to perform an assessment of the process safety culture within their institution in order to determine how faculty and students view process safety.more »
-
Process safety is at the heart of operation of many chemical processing companies. However, the Chemical Safety Board (CSB) has still documented over 800 investigations of process safety failures since the year 2000. While not all of these incidents were severe, some did lead to employee injuries or death and environmental harm. As a result, chemical engineering companies are increasingly dedicated to process safety through training programs and detailed vigilance as part of their operations practice. AIChE and OSHA also offer courses in process safety to help support the industry. These efforts illustrate the paramount importance that chemical engineering graduates have an appreciation and understanding of process safety as they transition from their degree program into industrial positions. Previous studies have shown that despite difficulties due to course load constraints, process safety has been incorporated into chemical engineering curriculum through either the addition of new courses, incorporation of the content within existing classes, or a combination of the two methods. A review performed in Process Safety Progress suggested that a key step for departments moving forward is to perform an assessment of the process safety culture within their institution in order to determine how faculty and students view process safety.more »
-
Process safety is at the heart of operation of many chemical processing companies. However, the Chemical Safety Board (CSB) has still documented over 800 investigations of process safety failures since the year 2000. While not all of these incidents were severe, some did lead to employee injuries or death and environmental harm. As a result, chemical engineering companies are increasingly dedicated to process safety through training programs and detailed vigilance as part of their operations practice. AIChE and OSHA also offer courses in process safety to help support the industry. These efforts illustrate the paramount importance that chemical engineering graduates have an appreciation and understanding of process safety as they transition from their degree program into industrial positions. Previous studies have shown that despite difficulties due to course load constraints, process safety has been incorporated into chemical engineering curriculum through either the addition of new courses, incorporation of the content within existing classes, or a combination of the two methods. A review performed in Process Safety Progress suggested that a key step for departments moving forward is to perform an assessment of the process safety culture within their institution in order to determine how faculty and students view process safety.more »