Title: Probabilistic simulation supports generalizable intuitive physics
How do people perform general-purpose physical reasoning across a variety of scenarios in everyday life? Across two stud ies with seven different physical scenarios, we asked participants to predict whether or where two objects will make contact. People achieved high accuracy and were highly consistent with each other in their predictions. We hypothesize that this robust generalization is a consequence of mental simulations of noisy physics. We designed an “intuitive physics engine” model to capture this generalizable simulation. We find that this model generalized in human-like ways to unseen stimuli and to a different query of predictions. We evaluated several state-of-the-art deep learning and scene feature models on the same task and found that they could not explain human predictions as well. This study provides evidence that human’s robust generalization in physics predictions are supported by a probabilistic simulation model, and suggests the need for structure in learned dynamics models.  more » « less
Award ID(s):
2121009
PAR ID:
10542746
Author(s) / Creator(s):
; ; ; ; ; ; ;
Publisher / Repository:
Cognitive Science Society
Date Published:
Format(s):
Medium: X
Location:
https://escholarship.org/content/qt93j3f86q/qt93j3f86q_noSplash_cc592224fe0e74bb9c5a779349e62558.pdf
Sponsoring Org:
National Science Foundation
More Like this
  1. The social media have been increasingly used for disaster management (DM) via providing real time data on a broad scale. For example, some smartphone applications (e.g. Disaster Alert and Federal Emergency Management Agency (FEMA) App) can be used to increase the efficiency of prepositioning supplies and to enhance the effectiveness of disaster relief efforts. To maximize utilities of these apps, it is imperative to have robust human behavior models in social networks with detailed expressions of individual decision-making processes and of the interactions among people. In this paper, we introduce a hierarchical human behavior model by associating extended Decision Field Theory (e-DFT) with the opinion formation and innovation diffusion models. Particularly, its expressiveness and validity are addressed in three ways. First, we estimate individual’s choice patterns in social networks by deriving people’s asymptotic choice probabilities within e-DFT. Second, by analyzing opinion formation models and innovation diffusion models in different types of social networks, the effects of neighbor’s opinions on people and their interactions are demonstrated. Finally, an agent-based simulation is used to trace agents’ dynamic behaviors in different scenarios. The simulated results reveal that the proposed models can be used to establish better disaster management strategies in natural disasters. 
    more » « less
  2. Heating, ventilation, and air conditioning (HVAC) systems account for a significant portion of energy consumption in developed countries. Accurately identifying their dynamics is crucial for developing effective controllers. However, it is challenging due to the system’s nonlinearities and variations across building types. Data-driven approaches have shown great promise in modeling the dynamics of HVAC systems, replacing traditional ordinary differential equations. However, relying solely on data can lead to poor generalization, particularly when the training data is limited or when unknown disturbances, such as weather conditions and occupant behavior, are present. Physics-informed machine learning (PIML) techniques have been developed for integrating physical principles into machine learning methods to improve the accuracy and data efficiency of modeling of dynamical systems. This paper investigates PIML techniques that incorporate three different physical properties: monotonicity, boundedness, and system structure. These models are benchmarked against physics-agnostic machine learning (PAML) approaches and the gray-box modeling technique to further highlight the performance of PIML models in terms of accuracy, robustness, and data efficiency for modeling HVAC systems from real data. Experimental data collected from a real-world HVAC system are used for systematically studying and analyzing thirteen gray-box, PAML, and PIML modeling techniques in different scenarios. Our results demonstrate that PIML models outperform PAML models in predicting the temperature dynamics of HVAC systems, especially when the training data is limited, unreliable, or noisy, with accurate and robust performance. Furthermore, we identify which physical properties are the most beneficial for enhancing machine learning performance for HVAC system identification. 
    more » « less
  3. Reliable generalization metrics are fundamental to the evaluation of machine learning models. Especially in high-stakes applications where labeled target data are scarce, evaluation of models’ generalization performance under distribution shift is a pressing need. We focus on two practical scenarios: (1) Before deployment, how to select the best model for unlabeled target data? (2) After deployment, how to monitor model performance under distribution shift? The central need in both cases is a reliable and label-free proxy metric. Yet existing proxy metrics, such as model confidence or accuracy-on-the-line, are often unreliable as they only assess model output while ignoring the internal mechanisms that produce them. We address this limitation by introducing a new perspective: using the inner workings of a model, i.e., circuits, as a predictive metric of generalization performance. Leveraging circuit discovery, we extract the causal interactions between internal representations as a circuit, from which we derive two metrics tailored to the two practical scenarios. (1) Before deployment, we introduce Dependency Depth Bias, which measures different models’ generalization capability on target data. (2) After deployment, we propose Circuit Shift Score, which predicts a model’s generalization under different distribution shifts. Across various tasks, both metrics demonstrate significantly improved correlation with generalization performance, outperforming existing proxies by an average of 13.4% and 34.1%, respectively. Our code is available at https://github.com/deep-real/GenCircuit. 
    more » « less
  4. While current vision algorithms excel at many challenging tasks, it is unclear how well they understand the physical dynamics of real-world environments. Here we introduce Physion, a dataset and benchmark for rigorously evaluating the ability to predict how physical scenarios will evolve over time. Our dataset features realistic simulations of a wide range of physical phenomena, including rigid and soft-body collisions, stable multi-object configurations, rolling, sliding, and projectile motion, thus providing a more comprehensive challenge than previous benchmarks. We used Physion to benchmark a suite of models varying in their architecture, learning objective, input-output structure, and training data. In parallel, we obtained precise measurements of human prediction behavior on the same set of scenarios, allowing us to directly evaluate how well any model could approximate human behavior. We found that vision algorithms that learn object-centric representations generally outperform those that do not, yet still fall far short of human performance. On the other hand, graph neural networks with direct access to physical state information both perform substantially better and make predictions that are more similar to those made by humans. These results suggest that extracting physical representations of scenes is the main bottleneck to achieving human-level and human-like physical understanding in vision algorithms. We have publicly released all data and code to facilitate the use of Physion to benchmark additional models in a fully reproducible manner, enabling systematic evaluation of progress towards vision algorithms that understand physical environments as robustly as people do. 
    more » « less
  5. We study the implicit bias of optimization in robust empirical risk minimization (robust ERM) and its connection with robust generalization. In classification settings under adversarial perturbations with linear models, we study what type of regularization should ideally be applied for a given perturbation set to improve (robust) generalization. We then show that the implicit bias of optimization in robust ERM can significantly affect the robustness of the model and identify two ways this can happen; either through the optimization algorithm or the architecture. We verify our predictions in simulations with synthetic data and experimentally study the importance of implicit bias in robust ERM with deep neural networks. 
    more » « less