More Like this

1. DIM: Dyadic Interaction Modeling for Social Behavior Generation

   https://doi.org/10.1007/978-3-031-72913-3_27  

   Tran, Minh; Chang, Di; Siniukov, Maksim; Soleymani, Mohammad (December 2024, Springer Nature Switzerland)

   Leonardis, Aleš; Ricci, Eliss; Roth, Stefan; Russakovsky, Olga; Sattler, Torsten; Varol, Gul (Ed.)

   Human-human communication is like a delicate dance where listeners and speakers concurrently interact to maintain conversational dynamics. Hence, an effective model for generating listener nonverbal behaviors requires understanding the dyadic context and interaction. In this paper, we present an effective framework for creating 3D facial motions in dyadic interactions. Existing work consider a listener as a reactive agent with reflexive behaviors to the speaker’s voice and facial motions. The heart of our framework is Dyadic Interaction Modeling (DIM), a pre-training approach that jointly models speakers’ and listeners’ motions through masking and contrastive learning to learn representations that capture the dyadic context. To enable the generation of non-deterministic behaviors, we encode both listener and speaker motions into discrete latent representations, through VQ-VAE. The pre-trained model is further fine-tuned for motion generation. Extensive experiments demonstrate the superiority of our framework in generating listener motions, establishing a new state-of-the-art according to the quantitative measures capturing the diversity and realism of generated motions. Qualitative results demonstrate the superior capabilities of the proposed approach in generating diverse and realistic expressions, eye blinks and head gestures. 

   more » « less
2. Physics-Guided Machine Learning for Scientific Discovery: An Application in Simulating Lake Temperature Profiles

   https://doi.org/10.1145/3447814  

   Jia, Xiaowei; Willard, Jared; Karpatne, Anuj; Read, Jordan S.; Zwart, Jacob A.; Steinbach, Michael; Kumar, Vipin (May 2021, ACM/IMS Transactions on Data Science)

   null (Ed.)

   Physics-based models are often used to study engineering and environmental systems. The ability to model these systems is the key to achieving our future environmental sustainability and improving the quality of human life. This article focuses on simulating lake water temperature, which is critical for understanding the impact of changing climate on aquatic ecosystems and assisting in aquatic resource management decisions. General Lake Model (GLM) is a state-of-the-art physics-based model used for addressing such problems. However, like other physics-based models used for studying scientific and engineering systems, it has several well-known limitations due to simplified representations of the physical processes being modeled or challenges in selecting appropriate parameters. While state-of-the-art machine learning models can sometimes outperform physics-based models given ample amount of training data, they can produce results that are physically inconsistent. This article proposes a physics-guided recurrent neural network model (PGRNN) that combines RNNs and physics-based models to leverage their complementary strengths and improves the modeling of physical processes. Specifically, we show that a PGRNN can improve prediction accuracy over that of physics-based models (by over 20% even with very little training data), while generating outputs consistent with physical laws. An important aspect of our PGRNN approach lies in its ability to incorporate the knowledge encoded in physics-based models. This allows training the PGRNN model using very few true observed data while also ensuring high prediction accuracy. Although we present and evaluate this methodology in the context of modeling the dynamics of temperature in lakes, it is applicable more widely to a range of scientific and engineering disciplines where physics-based (also known as mechanistic) models are used. 

   more » « less
3. Physics guided RNNs for modeling dynamical systems: A case study in simulating lake temperature profiles

   Jia, X.; Willard, J.; Karpatne, A.; Read, J.; Zwart, J.; Steinbach, M.; Kumar, V. (May 2019, Proceedings of the 2019 SIAM International Conference on Data Mining)

   This paper proposes a physics-guided recurrent neural network model (PGRNN) that combines RNNs and physics-based models to leverage their complementary strengths and improve the modeling of physical processes. Specifically, we show that a PGRNN can improve prediction accuracy over that of physical models, while generating outputs consistent with physical laws, and achieving good generalizability. Standard RNNs, even when producing superior prediction accuracy, often produce physically inconsistent results and lack generalizability. We further enhance this approach by using a pre-training method that leverages the simulated data from a physics-based model to address the scarcity of observed data. Although we present and evaluate this methodology in the context of modeling the dynamics of temperature in lakes, it is applicable more widely to a range of scientific and engineering disciplines where mechanistic (also known as process-based) models are used, e.g., power engineering, climate science, materials science, computational chemistry, and biomedicine. 

   more » « less
4. A hybrid stochastic model of the budding yeast cell cycle

   https://doi.org/10.1038/s41540-020-0126-z  

   Ahmadian, Mansooreh; Tyson, John J.; Peccoud, Jean; Cao, Yang (March 2020, npj Systems Biology and Applications)

   Abstract The growth and division of eukaryotic cells are regulated by complex, multi-scale networks. In this process, the mechanism of controlling cell-cycle progression has to be robust against inherent noise in the system. In this paper, a hybrid stochastic model is developed to study the effects of noise on the control mechanism of the budding yeast cell cycle. The modeling approach leverages, in a single multi-scale model, the advantages of two regimes: (1) the computational efficiency of a deterministic approach, and (2) the accuracy of stochastic simulations. Our results show that this hybrid stochastic model achieves high computational efficiency while generating simulation results that match very well with published experimental measurements. 

   more » « less
5. Hybrid Geometry/Property Autoencoders for Multi-Lattice Transitions

   Baldwin, M; Meisel, N. A.; McComb, C. (October 2023, 2023 International Solid Freeform Fabrication Symposium)

   Additive manufacturing has revolutionized structural optimization by enhancing component strength and reducing material requirements. One approach used to achieve these improvements is the application of multi-lattice structures. The performance of these structures heavily relies on the detailed design of mesostructural elements. Many current approaches use data-driven design to generate multi-lattice transition regions, making use of models that jointly address the geometry and properties of the mesostructures. However, it remains unclear whether the integration of mechanical properties into the data set for generating multi-lattice interpolations is beneficial beyond geometry alone. To address this issue, this work implements and evaluates a hybrid geometry/property machine learning model for generating multi-lattice transition regions. We compare the results of this hybrid model to results obtained using a geometry-only model. Our research determined that incorporating physical properties decreased the number of variables to address in the latent space, and therefore improves the ability of generative models for developing transition regions of multi-lattice structures. 

   more » « less