Understanding what sequence of steps are needed to complete a goal can help artificial intelligence systems reason about human activities. Past work in NLP has examined the task of goal-step inference for text. We introduce the visual analogue. We propose the Visual Goal-Step Inference (VGSI) task, where a model is given a textual goal and must choose which of four images represents a plausible step towards that goal. With a new dataset harvested from wikiHow consisting of 772,277 images representing human actions, we show that our task is challenging for state-of-the-art multimodal models. Moreover, the multimodal representation learned from our data can be effectively transferred to other datasets like HowTo100m, increasing the VGSI accuracy by 15 - 20%. Our task will facilitate multimodal reasoning about procedural events.
more »
« less
The rational selection of goal operations and the integration of search strategies with goal-driven marine autonomy
Intelligent physical systems as embodied cognitive systems must perform high-level reasoning while concurrently managing an underlying control architecture. The link between cognition and control must manage the problem of converting continuous values from the real world to symbolic representations (and back). To generate effective behaviors, reasoning must include a capacity to replan, acquire and update new information, detect and respond to anomalies, and perform various operations on system goals. But, these processes are not independent and need further exploration. This paper examines an agent’s choices when multiple goal operations co-occur and interact, and it establishes a method of choosing between them. We demonstrate the benefits and discuss the trade offs involved with this and show positive results in a dynamic marine search task.
more »
« less
- Award ID(s):
- 1849131
- NSF-PAR ID:
- 10352601
- Date Published:
- Journal Name:
- Proceedings of the Ninth Annual Conference on Advances in Cognitive Systems
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Process safety is at the heart of operation of many chemical processing companies. However, the Chemical Safety Board (CSB) has still documented over 800 investigations of process safety failures since the year 2000. While not all of these incidents were severe, some did lead to employee injuries or death and environmental harm. As a result, chemical engineering companies are increasingly dedicated to process safety through training programs and detailed vigilance as part of their operations practice. AIChE and OSHA also offer courses in process safety to help support the industry. These efforts illustrate the paramount importance that chemical engineering graduates have an appreciation and understanding of process safety as they transition from their degree program into industrial positions. Previous studies have shown that despite difficulties due to course load constraints, process safety has been incorporated into chemical engineering curriculum through either the addition of new courses, incorporation of the content within existing classes, or a combination of the two methods. A review performed in Process Safety Progress suggested that a key step for departments moving forward is to perform an assessment of the process safety culture within their institution in order to determine how faculty and students view process safety. An issue with completing this task is the lack of assessment tools that can be used to determine how students are developing their understanding of process safety decision making. This observation led to the development of the Engineering Process Safety Research Instrument (EPSRI). This instrument is modeled after the Defining Issues Test version 2 (DIT2) and the Engineering Ethical Reasoning Instrument (EERI). Similar to these instruments, the EPSRI provides dilemmas, three decisions, and 12 additional considerations that individuals must rate based on their relative importance to their decision making process. The dilemmas developed in the EPSRI are based on case studies and investigations from process safety failures that have occurred in industry to provide a realistic context for the decision making decisions that engineers may be faced with upon employment. The considerations provided after the scenario are derived to reflect pre-conventional, conventional, and post-conventional decision making thinking as described by Kohlberg’s Moral Development Theory. Pre-conventional decision making thinking focuses particularly on what is right/wrong or good/bad from an individual level, whereas post-conventional thinking seeks to determine what is correct from moral and value perspectives at the society level. This WIP paper describes the content validity study conducted while developing the EPSRI. Dilemmas were examined by context experts including professionals in the process industry, chemical engineering departments, and learning sciences field. Content experts reviewed the dilemmas and determined whether they represented accurate examples of process safety decision making that individuals may face in real-world engineering settings. The experts also reviewed the 12 considerations for each dilemma for their accuracy in capturing pre-conventional, conventional and post-conventional thinking. This work represents the first step in the overall instrument validation that will take place over the next academic year.more » « less
-
Process safety is at the heart of operation of many chemical processing companies. However, the Chemical Safety Board (CSB) has still documented over 800 investigations of process safety failures since the year 2000. While not all of these incidents were severe, some did lead to employee injuries or death and environmental harm. As a result, chemical engineering companies are increasingly dedicated to process safety through training programs and detailed vigilance as part of their operations practice. AIChE and OSHA also offer courses in process safety to help support the industry. These efforts illustrate the paramount importance that chemical engineering graduates have an appreciation and understanding of process safety as they transition from their degree program into industrial positions. Previous studies have shown that despite difficulties due to course load constraints, process safety has been incorporated into chemical engineering curriculum through either the addition of new courses, incorporation of the content within existing classes, or a combination of the two methods. A review performed in Process Safety Progress suggested that a key step for departments moving forward is to perform an assessment of the process safety culture within their institution in order to determine how faculty and students view process safety. An issue with completing this task is the lack of assessment tools that can be used to determine how students are developing their understanding of process safety decision making. This observation led to the development of the Engineering Process Safety Research Instrument (EPSRI). This instrument is modeled after the Defining Issues Test version 2 (DIT2) and the Engineering Ethical Reasoning Instrument (EERI). Similar to these instruments, the EPSRI provides dilemmas, three decisions, and 12 additional considerations that individuals must rate based on their relative importance to their decision making process. The dilemmas developed in the EPSRI are based on case studies and investigations from process safety failures that have occurred in industry to provide a realistic context for the decision making decisions that engineers may be faced with upon employment. The considerations provided after the scenario are derived to reflect pre-conventional, conventional, and post-conventional decision making thinking as described by Kohlberg’s Moral Development Theory. Pre-conventional decision making thinking focuses particularly on what is right/wrong or good/bad from an individual level, whereas post-conventional thinking seeks to determine what is correct from moral and value perspectives at the society level. This WIP paper describes the content validity study conducted while developing the EPSRI. Dilemmas were examined by context experts including professionals in the process industry, chemical engineering departments, and learning sciences field. Content experts reviewed the dilemmas and determined whether they represented accurate examples of process safety decision making that individuals may face in real-world engineering settings. The experts also reviewed the 12 considerations for each dilemma for their accuracy in capturing pre-conventional, conventional and post-conventional thinking. This work represents the first step in the overall instrument validation that will take place over the next academic year.more » « less
-
Process safety is at the heart of operation of many chemical processing companies. However, the Chemical Safety Board (CSB) has still documented over 800 investigations of process safety failures since the year 2000. While not all of these incidents were severe, some did lead to employee injuries or death and environmental harm. As a result, chemical engineering companies are increasingly dedicated to process safety through training programs and detailed vigilance as part of their operations practice. AIChE and OSHA also offer courses in process safety to help support the industry. These efforts illustrate the paramount importance that chemical engineering graduates have an appreciation and understanding of process safety as they transition from their degree program into industrial positions. Previous studies have shown that despite difficulties due to course load constraints, process safety has been incorporated into chemical engineering curriculum through either the addition of new courses, incorporation of the content within existing classes, or a combination of the two methods. A review performed in Process Safety Progress suggested that a key step for departments moving forward is to perform an assessment of the process safety culture within their institution in order to determine how faculty and students view process safety. An issue with completing this task is the lack of assessment tools that can be used to determine how students are developing their understanding of process safety decision making. This observation led to the development of the Engineering Process Safety Research Instrument (EPSRI). This instrument is modeled after the Defining Issues Test version 2 (DIT2) and the Engineering Ethical Reasoning Instrument (EERI). Similar to these instruments, the EPSRI provides dilemmas, three decisions, and 12 additional considerations that individuals must rate based on their relative importance to their decision making process. The dilemmas developed in the EPSRI are based on case studies and investigations from process safety failures that have occurred in industry to provide a realistic context for the decision making decisions that engineers may be faced with upon employment. The considerations provided after the scenario are derived to reflect pre-conventional, conventional, and post-conventional decision making thinking as described by Kohlberg’s Moral Development Theory. Pre-conventional decision making thinking focuses particularly on what is right/wrong or good/bad from an individual level, whereas post-conventional thinking seeks to determine what is correct from moral and value perspectives at the society level. This WIP paper describes the content validity study conducted while developing the EPSRI. Dilemmas were examined by context experts including professionals in the process industry, chemical engineering departments, and learning sciences field. Content experts reviewed the dilemmas and determined whether they represented accurate examples of process safety decision making that individuals may face in real-world engineering settings. The experts also reviewed the 12 considerations for each dilemma for their accuracy in capturing pre-conventional, conventional and post-conventional thinking. This work represents the first step in the overall instrument validation that will take place over the next academic year.more » « less
-
Process safety is at the heart of operation of many chemical processing companies. However, the Chemical Safety Board (CSB) has still documented over 800 investigations of process safety failures since the year 2000. While not all of these incidents were severe, some did lead to employee injuries or death and environmental harm. As a result, chemical engineering companies are increasingly dedicated to process safety through training programs and detailed vigilance as part of their operations practice. AIChE and OSHA also offer courses in process safety to help support the industry. These efforts illustrate the paramount importance that chemical engineering graduates have an appreciation and understanding of process safety as they transition from their degree program into industrial positions. Previous studies have shown that despite difficulties due to course load constraints, process safety has been incorporated into chemical engineering curriculum through either the addition of new courses, incorporation of the content within existing classes, or a combination of the two methods. A review performed in Process Safety Progress suggested that a key step for departments moving forward is to perform an assessment of the process safety culture within their institution in order to determine how faculty and students view process safety. An issue with completing this task is the lack of assessment tools that can be used to determine how students are developing their understanding of process safety decision making. This observation led to the development of the Engineering Process Safety Research Instrument (EPSRI). This instrument is modeled after the Defining Issues Test version 2 (DIT2) and the Engineering Ethical Reasoning Instrument (EERI). Similar to these instruments, the EPSRI provides dilemmas, three decisions, and 12 additional considerations that individuals must rate based on their relative importance to their decision making process. The dilemmas developed in the EPSRI are based on case studies and investigations from process safety failures that have occurred in industry to provide a realistic context for the decision making decisions that engineers may be faced with upon employment. The considerations provided after the scenario are derived to reflect pre-conventional, conventional, and post-conventional decision making thinking as described by Kohlberg’s Moral Development Theory. Pre-conventional decision making thinking focuses particularly on what is right/wrong or good/bad from an individual level, whereas post-conventional thinking seeks to determine what is correct from moral and value perspectives at the society level. This WIP paper describes the content validity study conducted while developing the EPSRI. Dilemmas were examined by context experts including professionals in the process industry, chemical engineering departments, and learning sciences field. Content experts reviewed the dilemmas and determined whether they represented accurate examples of process safety decision making that individuals may face in real-world engineering settings. The experts also reviewed the 12 considerations for each dilemma for their accuracy in capturing pre-conventional, conventional and post-conventional thinking. This work represents the first step in the overall instrument validation that will take place over the next academic year.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
