More Like this

1. Learning algebraic representation for systematic generalization in abstract reasoning

   Zhang, C.; Xie, S.; Jia B.; Wu, Y. N.; Zhu, S.-C.; Zhu, Y. (January 2022, European Conference on Computer Vision (ECCV 2022))

   Is intelligence realized by connectionist or classicist? While connectionist approaches have achieved superhuman performance, there has been growing evidence that such task-specific superiority is particularly fragile in systematic generalization. This observation lies in the central debate between connectionist and classicist, wherein the latter continually advocates an algebraic treatment in cognitive architectures. In this work, we follow the classicist’s call and propose a hybrid approach to improve systematic generalization in reasoning. Specifically, we showcase a prototype with algebraic representation for the abstract spatial-temporal reasoning task of Raven’s Progressive Matrices (RPM) and present the ALgebra-Aware Neuro-Semi-Symbolic (ALANS) learner. The ALANS learner is motivated by abstract algebra and the representation theory. It consists of a neural visual perception frontend and an algebraic abstract reasoning backend: the frontend summarizes the visual information from object-based representation, while the backend transforms it into an algebraic structure and induces the hidden operator on the fly. The induced operator is later executed to predict the answer’s representation, and the choice most similar to the prediction is selected as the solution. Extensive experiments show that by incorporating an algebraic treatment, the ALANS learner outperforms various pure connectionist models in domains requiring systematic generalization. We further show the generative nature of the learned algebraic representation; it can be decoded by isomorphism to generate an answer. 

   more » « less
2. Learning Factorized Multimodal Representations

   Tsai, Yao-Hung Hubert; Liang, Paul Pu; Zadeh, Amir; Morency, Louis-Philippe; Salakhutdinov, Ruslan (February 2019, International Conference on Representation Learning)

   Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information. Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) models must learn the complex intra-modal and cross-modal interactions for prediction and 2) models must be robust to unexpected missing or noisy modalities during testing. In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels. We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-specific generative factors. Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction. Modality-specific generative factors are unique for each modality and contain the information required for generating data. Experimental results show that our model is able to learn meaningful multimodal representations that achieve state-of-the-art or competitive performance on six multimodal datasets. Our model demonstrates flexible generative capabilities by conditioning on independent factors and can reconstruct missing modalities without significantly impacting performance. Lastly, we interpret our factorized representations to understand the interactions that influence multimodal learning. 

   more » « less
3. Analyzing Machine Learning Models that Predict Mental Illnesses from Social Media Text

   Weerasinghe, Janith; Morales, Kediel; Greenstadt, Rachel (July 2018, Privacy Enhancing Technologies Symposium)

   Previous studies, both in psychology and linguistics, have shown that individuals with mental illnesses show deviations from normal language use, that these differences can be used to make predictions, and used as a diagnostic tool. Recent studies have shown that machine learning can be used to predict people with mental illnesses based on their writing. However, little attention is paid to the interpretability of the machine learning models. In this talk we will describe our analysis of the machine learning models, the different language patterns that distinguish individuals having mental illnesses from a control group, and the associated privacy concerns. We use a dataset of Tweets that are collected from users who reported a diagnosis of a mental illnesses on Twitter. Given the self-reported nature of the dataset, it is possible that some of these individuals are actively talking about their mental illness on social media. We investigated if the machine learning models are detecting the active mentions of the mental illness or if they are detecting more complex language patterns. We then conducted a feature analysis by creating feature vectors using word unigrams, part of speech tags and word clusters and used feature importance measures and statistical methods to identify important features. This analysis serves two purposes: to understand the machine learning model, and to discover language patterns that would help in identifying people with mental illnesses. Finally, we conducted a qualitative analysis of the misclassifications to understand the potential causes for the misclassifications. 

   more » « less
4. Mental models use common neural spatial structure for spatial and abstract content

   https://doi.org/10.1038/s42003-019-0740-8  

   Alfred, Katherine L.; Connolly, Andrew C.; Cetron, Joshua S.; Kraemer, David J. M. (January 2020, Communications Biology)

   Abstract Mental models provide a cognitive framework allowing for spatially organizing information while reasoning about the world. However, transitive reasoning studies often rely on perception of stimuli that contain visible spatial features, allowing the possibility that associated neural representations are specific to inherently spatial content. Here, we test the hypothesis that neural representations of mental models generated through transitive reasoning rely on a frontoparietal network irrespective of the spatial nature of the stimulus content. Content within three models ranges from expressly visuospatial to abstract. All mental models participants generated were based on inferred relationships never directly observed. Here, using multivariate representational similarity analysis, we show that patterns representative of mental models were revealed in both superior parietal lobule and anterior prefrontal cortex and converged across stimulus types. These results support the conclusion that, independent of content, transitive reasoning using mental models relies on neural mechanisms associated with spatial cognition. 

   more » « less
5. SPNets: Human-like Navigation Behaviors with Uncertain Goals

   https://doi.org/10.1145/3424636.3426911  

   Sohre, Nicholas; Guy, Stephen J. (October 2020, SPNets: Human-like Navigation Behaviors with Uncertain Goals)

   null (Ed.)

   Most path planning techniques use exact, global information of the environment to make optimal or near-optimal plans. In contrast, humans navigate using only local information, which they must augment with their understanding of typical building layouts to guess what lies ahead, while integrating what they have seen already to form mental representations of building structure. Here, we propose Scene Planning Networks (SPNets), a neural network based approach for formulating the long-range navigation problem as a series of local decisions similar to what humans face when navigating. Agents navigating using SPNets build additive neural representations of previous observations to understand local obstacle structure, and use a network-based planning approach to plan the next steps towards a fuzzy goal region. Our approach reproduces several important aspects of human behavior that are not captured by either full global planning or simple local heuristics. 

   more » « less