skip to main content


The NSF Public Access Repository (NSF-PAR) system and access will be unavailable from 11:00 PM ET on Thursday, May 23 until 2:00 AM ET on Friday, May 24 due to maintenance. We apologize for the inconvenience.

Title: Learning algebraic representation for systematic generalization in abstract reasoning
Is intelligence realized by connectionist or classicist? While connectionist approaches have achieved superhuman performance, there has been growing evidence that such task-specific superiority is particularly fragile in systematic generalization. This observation lies in the central debate between connectionist and classicist, wherein the latter continually advocates an algebraic treatment in cognitive architectures. In this work, we follow the classicist’s call and propose a hybrid approach to improve systematic generalization in reasoning. Specifically, we showcase a prototype with algebraic representation for the abstract spatial-temporal reasoning task of Raven’s Progressive Matrices (RPM) and present the ALgebra-Aware Neuro-Semi-Symbolic (ALANS) learner. The ALANS learner is motivated by abstract algebra and the representation theory. It consists of a neural visual perception frontend and an algebraic abstract reasoning backend: the frontend summarizes the visual information from object-based representation, while the backend transforms it into an algebraic structure and induces the hidden operator on the fly. The induced operator is later executed to predict the answer’s representation, and the choice most similar to the prediction is selected as the solution. Extensive experiments show that by incorporating an algebraic treatment, the ALANS learner outperforms various pure connectionist models in domains requiring systematic generalization. We further show the generative nature of the learned algebraic representation; it can be decoded by isomorphism to generate an answer.  more » « less
Award ID(s):
Author(s) / Creator(s):
; ; ; ; ;
Date Published:
Journal Name:
European Conference on Computer Vision (ECCV 2022)
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Virtual content instability caused by device pose tracking error remains a prevalent issue in markerless augmented reality (AR), especially on smartphones and tablets. However, when examining environments which will host AR experiences, it is challenging to determine where those instability artifacts will occur; we rarely have access to ground truth pose to measure pose error, and even if pose error is available, traditional visualizations do not connect that data with the real environment, limiting their usefulness. To address these issues we present SiTAR (Situated Trajectory Analysis for Augmented Reality), the first situated trajectory analysis system for AR that incorporates estimates of pose tracking error. We start by developing the first uncertainty-based pose error estimation method for visual-inertial simultaneous localization and mapping (VI-SLAM), which allows us to obtain pose error estimates without ground truth; we achieve an average accuracy of up to 96.1% and an average FI score of up to 0.77 in our evaluations on four VI-SLAM datasets. Next, we present our SiTAR system, implemented for ARCore devices, combining a backend that supplies uncertainty-based pose error estimates with a frontend that generates situated trajectory visualizations. Finally, we evaluate the efficacy of SiTAR in realistic conditions by testing three visualization techniques in an in-the-wild study with 15 users and 13 diverse environments; this study reveals the impact both environment scale and the properties of surfaces present can have on user experience and task performance. 
    more » « less
  2. Abstract

    Engineering design involves intensive visual-spatial reasoning, and engineers depend upon external representation to develop concepts during idea generation. Previous research has not explored how our visual representation skills influence our idea generation effectiveness. A designer’s deficit in sketching skills could create a need for increased focus on the task of visual representation reducing cognitive resources available for the task at hand — generating concept. Further, this effect could be compounded if designers believed that their sketching skill would be evaluated or judged by their peers. This evaluation apprehension could cause additional mental workload distracting from the production of idea generation.

    The goal of this study is to investigate and better understand the relationship between designers’ sketching skills and idea generation abilities. In this paper, we present preliminary results of the relationship between independent measures of sketching skill and idea generation ability from an entry-level engineering design and graphics course. During data collection, task instructions were given in two ways to independent groups: one group was instructed upfront that sketching would be evaluated, while the second group was kept blind to the sketch evaluation. In this paper, we also examine the potential priming effects of sketch quality evaluation apprehension on idea generation productivity. The results show that sketching quality and idea quantity are largely independent, and that the priming effects of sketch evaluation instructions are small to negligible on idea generation productivity.

    more » « less
  3. Abstract

    Human reasoning is grounded in an ability to identify highly abstract commonalities governing superficially dissimilar visual inputs. Recent efforts to develop algorithms with this capacity have largely focused on approaches that require extensive direct training on visual reasoning tasks, and yield limited generalization to problems with novel content. In contrast, a long tradition of research in cognitive science has focused on elucidating the computational principles underlying human analogical reasoning; however, this work has generally relied on manually constructed representations. Here we present visiPAM (visual Probabilistic Analogical Mapping), a model of visual reasoning that synthesizes these two approaches. VisiPAM employs learned representations derived directly from naturalistic visual inputs, coupled with a similarity-based mapping operation derived from cognitive theories of human reasoning. We show that without any direct training, visiPAM outperforms a state-of-the-art deep learning model on an analogical mapping task. In addition, visiPAM closely matches the pattern of human performance on a novel task involving mapping of 3D objects across disparate categories.

    more » « less
  4. We develop an algebraic framework for sequential data assimilation of partially observed dynamical systems. In this framework, Bayesian data assimilation is embedded in a nonabelian operator algebra, which provides a representation of observables by multiplication operators and probability densities by density operators (quantum states). In the algebraic approach, the forecast step of data assimilation is represented by a quantum operation induced by the Koopman operator of the dynamical system. Moreover, the analysis step is described by a quantum effect, which generalizes the Bayesian observational update rule. Projecting this formulation to finite-dimensional matrix algebras leads to computational schemes that are i) automatically positivity-preserving and ii) amenable to consistent data-driven approximation using kernel methods for machine learning. Moreover, these methods are natural candidates for implementation on quantum computers. Applications to the Lorenz 96 multiscale system and the El Niño Southern Oscillation in a climate model show promising results in terms of forecast skill and uncertainty quantification.

    more » « less
  5. Abstract

    In this study, we analyzed students’ reasoning and explanations of friction concepts before and after engaging in guided experimentation with visuohaptic (VH) simulations. The VH experimentation included two affordances: visual cues and haptic feedback. Specifically, we analyzed the outcomes of two treatment groups with different sequences of affordance introduction. The first treatment group started with visual cues, with haptic feedback added later, while the second treatment group started with haptic feedback and added the visual cues later. We recruited 48 students who had previously taken at least one physics course. Participants completed a pre‐ and posttest assessment, which included both procedural and conceptual questions about friction before and after the guided experimentation task. The results show that the participants from both treatment groups benefited from using VH simulations. Both treatment groups showed statistically significant pre/post improvements in their understanding of friction. Moreover, both treatment groups showed a statistically significant increase in the conceptual understanding of friction concepts from pretest to posttest with moderate to strong effect sizes. Implications for laboratory instruction are also discussed.

    more » « less