An official website of the United States government
Continual learning is a setting where machine learning models learn novel concepts from continuously shifting training data, while simultaneously avoiding degradation of knowledge on previously seen classes which may disappear from the training data for extended periods of time (a phenomenon known as the catastrophic forgetting problem). Current approaches for continual learning of a single expanding task (aka class-incremental continual learning) require extensive rehearsal of previously seen data to avoid this degradation of knowledge. Unfortunately, rehearsal comes at a cost to memory, and it may also violate data-privacy. Instead, we explore combining knowledge distillation and parameter regularization in new ways to achieve strong continual learning performance without rehearsal. Specifically, we take a deep dive into common continual learning techniques: prediction distillation, feature distillation, L2 parameter regularization, and EWC parameter regularization. We first disprove the common assumption that parameter regularization techniques fail for rehearsal-free continual learning of a single, expanding task. Next, we explore how to leverage knowledge from a pre-trained model in rehearsal-free continual learning and find that vanilla L2 parameter regularization outperforms EWC parameter regularization and feature distillation. Finally, we explore the recently popular ImageNet-R benchmark, and show that L2 parameter regularization implemented in self-attention blocks of a ViT transformer outperforms recent popular prompting for continual learning methods.
more »
« less
This content will become publicly available on December 10, 2026
Using Incremental Learning With Rehearsal to Enhance Global Collapse Prediction Machine Learning Models Across Diverse Steel Building Datasets
Generalization of machine learning (ML) surrogate models across distinct databases is underexplored, despite being crucial as retraining the entire model every time new data become available is inefficient. This study proposes an incremental learning methodology to improve ML models' prediction of seismic collapse of steel moment‐resisting frames (SMRFs) across distinct datasets. Three boosting algorithms, XGBoost, LightGBM, and CatBoost, were trained on a source dataset to generate surrogate ML models that can predict the SMRF's seismic response. Thereafter, the ML models were used to predict the response on a new (target) dataset of SMRFs that differ in geometric dimensions and design approaches. Initially, boosting models trained on one dataset performed poorly on another dataset, even if the datasets displayed similar characteristics and consistent feature importance rankings. Incorporation of incremental learning improved the prediction on the target dataset, but introduced catastrophic forgetting that reduced the effectiveness of the ML model on the source dataset, a problem mitigated with a rehearsal strategy. Incremental learning with rehearsal yields results comparable to those obtained by fully retraining with both source and target datasets, resulting in an effective method for ML transferability, without having to retrain entire databases and without reducing the effectiveness of ML models on the source database.
more »
« less
- Award ID(s):
- 2121169
- PAR ID:
- 10656312
- Publisher / Repository:
- John Wiley & Sons
- Date Published:
- Journal Name:
- The Structural Design of Tall and Special Buildings
- Volume:
- 34
- Issue:
- 17
- ISSN:
- 1541-7794
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Machine learning (ML) techniques were generated for different information levels to identify the minimum set of system parameters required for predicting collapse and maximum interstory drift (SDR_max) of steel moment resisting frame (SMRF) buildings. Five baseline modern SMRFs were evaluated under seismic loading with varying system and ground motion (GM) parameters to generate a database. Classification and regression-based ML models were tested at three system information levels to predict collapse and SDR_max, respectively. The ML predictions were mainly controlled by GM parameters and were relatively insensitive to system parameters defining nonlinear behavior, such as spring backbone curve features.more » « less
-
Thermal conductivity (TC) is greatly influenced by the working temperature, microstructures, thermal processing (heat treatment) history and the composition of alloys. Due to computational costs and lengthy experimental procedures, obtaining the thermal conductivity for novel alloys, particularly parts made with additive manufacturing, is difficult and it is almost impossible to optimize the compositional space for an absolute targeted value of thermal conductivity. To address these difficulties, a machine learning method is explored to predict the TC of additive manufactured alloys. To accomplish this, an extensive thermal conductivity dataset for additively manufactured alloys was generated for several AM alloy families (nickel, copper, iron, cobalt-based) over various temperatures (300–1273 K). This unique dataset was used in training and validating machine learning models. Among the five different regression machine learning models trained with the dataset, extreme gradient boosting performs the best as compared with other models with an R2 score of 0.99. Furthermore, the accuracy of this model was tested using Inconel 718 and GRCop-42 fabricated with laser powder bed fusion-based additive manufacture, which have never been observed by the extreme gradient boosting model, and a good match between the experimental results and machine learning prediction was observed. The average mean error in predicting the thermal conductivity of Inconel 718 and GRCop-42 at different temperatures was 3.9% and 2.08%, respectively. This paper demonstrates that the thermal conductivity of novel AM alloys could be predicted quickly based on the dataset and the ML model.more » « less
-
Computer vision models suffer from a phenomenon known as catastrophic forgetting when learning novel concepts from continuously shifting training data. Typical solutions for this continual learning problem require extensive rehearsal of previously seen data, which increases memory costs and may violate data privacy. Recently, the emergence of large-scale pre-trained vision transformer models has enabled prompting approaches as an alternative to data-rehearsal. These approaches rely on a key-query mechanism to generate prompts and have been found to be highly resistant to catastrophic forgetting in the well-established rehearsal-free continual learning setting. However, the key mechanism of these methods is not trained end-to-end with the task sequence. Our experiments show that this leads to a reduction in their plasticity, hence sacrificing new task accuracy, and inability to benefit from expanded parameter capacity. We instead propose to learn a set of prompt components which are assembled with input-conditioned weights to produce input-conditioned prompts, resulting in a novel attention-based end-to-end key-query scheme. Our experiments show that we outperform the current SOTA method DualPrompt on established benchmarks by as much as 4.5% in average final accuracy. We also outperform the state of art by as much as 4.4% accuracy on a continual learning benchmark which contains both class-incremental and domain-incremental task shifts, corresponding to many practical settings.more » « less
-
Abstract Sheet metal stamped and welded assemblies, such as the ones used in automotive body-in-white (BIW) structures, have various sources of manufacturing variations during stamping and assembly processes. One of the major contributors to these variations is the springback on clamping release due to elastic recovery. Mitigating these variations requires expert knowledge of mechanical behavior, tooling, and process design. No analytical models can be used for the variety of geometries. Nonlinear FEA is also being used to predict springback, but it is time-consuming and requires specialized expertise, which makes it difficult to use in design exploration. Machine learning holds the promise of democratizing such complex analyses. This paper presents several case studies for data curation/generation, ML training, and validation. The prediction and quantification of the effects of springback are done on two levels: (i) low granularity, which involves predicting variations in certain parameters that are critical to measuring and understand spring back, and (ii) high granularity, predicting the shape of the component while taking into account the effects of springback and the stresses in the components. The data required to train, test, and validate the ML models were generated previously using an automated, integrated multi-stage simulation approach that was necessary to produce large datasets. Stamping simulations were validated against NUMISHEET benchmarks and also compared to test results published by other researchers. Subsequently, machine learning models were trained on the curated dataset to predict 2D stamped component shapes after springback and stress distributions across these shapes. For the assembly dataset, parameters such as unconstrained planar minimum zone magnitudes, angles between component planes, and twist angles are predicted using machine learning models, including linear and polynomial regression, decision trees, gradient boosting regression, support vector regression, and fully connected neural networks, and compared for their performance using consistent metrics. Hyper-parameter tuning is performed to optimize model performance, with artificial neural networks demonstrating promising capabilities in understanding variations in forming and multi-stage assembly processes.more » « less
