An official website of the United States government
The rising frequency of natural disasters demands efficient and accurate structural damage assessments to ensure public safety and expedite recovery. Human error, inconsistent standards, and safety risks limit traditional visual inspections by engineers. Although UAVs and AI have advanced post-disaster assessments, they still lack the expert knowledge and decision-making judgment of human inspectors. This study explores how expertise shapes human–building interaction during disaster inspections by using eye tracking technology to capture the gaze patterns of expert and novice inspectors. A controlled, screen-based inspection method was employed to safely gather data, which was then used to train a machine learning model for saliency map prediction. The results highlight significant differences in visual attention between experts and novices, providing valuable insights for future inspection strategies and training novice inspectors. By integrating human expertise with automated systems, this research aims to improve the accuracy and reliability of post-disaster structural assessments, fostering more effective human–machine collaboration in disaster response efforts.
more »
« less
Dynamic Fusion of Eye Movement Data and Verbal Narrations in Knowledge-rich Domains
We propose to jointly analyze experts' eye movements and verbal narrations to discover important and interpretable knowledge patterns to better understand their decision-making processes. The discovered patterns can further enhance data-driven statistical models by fusing experts' domain knowledge to support complex human-machine collaborative decision-making. Our key contribution is a novel dynamic Bayesian nonparametric model that assigns latent knowledge patterns into key phases involved in complex decision-making. Each phase is characterized by a unique distribution of word topics discovered from verbal narrations and their dynamic interactions with eye movement patterns, indicating experts' special perceptual behavior within a given decision-making stage. A new split-merge-switch sampler is developed to efficiently explore the posterior state space with an improved mixing rate. Case studies on diagnostic error prediction and disease morphology categorization help demonstrate the effectiveness of the proposed model and discovered knowledge patterns.
more »
« less
- Award ID(s):
- 1814450
- PAR ID:
- 10253145
- Date Published:
- Journal Name:
- Advances in neural information processing systems
- Volume:
- 33
- ISSN:
- 1049-5258
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Simple random negative sampling is a technique used to enhance decision-making in sequential models with numerous potential negative instances, like recommender systems. However, it ignores the patterns that can be discovered in complex sequences to select the most informative negative samples. In this paper, we address this challenge by introducing a Neighborhood-Aware Negative Sampling (NANS) technique in the context of student knowledge modeling (KM) and behavior modeling (BM). In the education domain, KM quantifies student knowledge based on past performance, while BM focuses on behaviors like student preferences of questions. With the vast number of problems to choose from and the intricate relationship between student knowledge and behavior, selecting the proper negative samples becomes a notable challenge in this problem. NANS, along with our proposed multi-objective, multi-task sequential model for KM and BM, NANS-KoBeM frames the simultaneous modeling of student knowledge and question selection as a multi-task learning problem with dual objectives: predicting students’ performance and their question selections.more » « less
-
Abstract Recent large-scale societal disruptions, from the COVID-19 pandemic to intensifying wildfires and weather events, reveal the importance of transforming governance systems so they can address complex, transboundary, and rapidly evolving crises. Yet current knowledge of the decision-making dynamics that yield transformative governance remains scant. Studies typically focus on the aggregate outputs of government decisions, while overlooking their micro-level underpinnings. This is a key oversight because drivers of policy change, such as learning or competition, are prosecuted by people rather than organizations. We respond to this knowledge gap by introducing a new analytical lens for understanding policymaking, aimed at uncovering how characteristics of decision-makers and the structure of their relationships affect their likelihood of effectuating transformative policy responses. This perspective emphasizes the need for a more dynamic and relational view on urban governance in the context of transformation.more » « less
-
Traditional models of decision making under uncertainty explain human behavior in simple situations with a minimal set of alternatives and attributes. Some of them, such as prospect theory, have been proven successful and robust in such simple situations. Yet, less is known about the preference formation during decision making in more complex cases. Furthermore, it is generally accepted that attention plays a role in the decision process but most theories make simplifying assumptions about where attention is deployed. In this study, we replace these assumptions by measuring where humans deploy overt attention, i.e. where they fixate. To assess the influence of task complexity, participants perform two tasks. The simpler of the two requires participants to choose between two alternatives with two attributes each (four items to consider). The more complex one requires a choice between four alternatives with four attributes each (16 items to consider). We then compare a large set of model classes, of different levels of complexity, by considering the dynamic interactions between uncertainty, attention and pairwise comparisons between attribute values. The task of all models is to predict what choices humans make, using the sequence of observed eye movements for each participant as input to the model. We find that two models outperform all others. The first is the two-layer leaky accumulator which predicts human choices on the simpler task better than any other model. We call the second model, which is introduced in this study, TNPRO. It is modified from a previous model from management science and designed to deal with highly complex decision problems. Our results show that this model performs well in the simpler of our two tasks (second best, after the accumulator model) and best for the complex task. Our results suggest that, when faced with complex choice problems, people prefer to accumulate preference based on attention-guided pairwise comparisons.more » « less
-
This paper investigates the decision-making outcomes and cognitive-physical load implications of integrating a Building Information Modeling-driven Augmented Reality (AR) system into retrofitting design and how movement is best leveraged to understand daylighting impacts. We conducted a study with 128 non-expert participants, who were asked to choose a window facade to improve an interior space. We found no significant difference in the overall decision-making outcome between those who used an AR tool or a conventional desktop approach and that greater eye movement in AR was related to non-experts better balancing the complicated impacts facades have on daylight, aesthetics, and energy.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
