An official website of the United States government
Machine learning (ML)-based data-driven methods have promoted the progress of modeling in many engineering domains. These methods can achieve high prediction and generalization performance for large, high-quality datasets. However, ML methods can yield biased predictions if the observed data (i.e., response variable y) are corrupted by outliers. This paper addresses this problem with a novel, robust ML approach that is formulated as an optimization problem by coupling locally weighted least-squares support vector machines for regression (LWLS-SVMR) with one weight function. The weight is a function of residuals and allows for iteration within the proposed approach, significantly reducing the negative interference of outliers. A new efficient hybrid algorithm is developed to solve the optimization problem. The proposed approach is assessed and validated by comparison with relevant ML approaches on both one-dimensional simulated datasets corrupted by various outliers and multi-dimensional real-world engineering datasets, including datasets used for predicting the lateral strength of reinforced concrete (RC) columns, the fuel consumption of automobiles, the rising time of a servomechanism, and dielectric breakdown strength. Finally, the proposed method is applied to produce a data-driven solver for computational mechanics with a nonlinear material dataset corrupted by outliers. The results all show that the proposed method is robust against non-extreme and extreme outliers and improves the predictive performance necessary to solve various engineering problems.
more »
« less
In Search of netUnicorn: A Data-Collection Platform to Develop Generalizable ML Models for Network Security Problems
The remarkable success of the use of machine learning-based solutions for network security problems has been impeded by the developed ML models’ inability to maintain efficacy when used in different network environments exhibiting different network behaviors. This issue is commonly referred to as the generalizability problem of ML models. The community has recognized the critical role that training datasets play in this context and has developed various techniques to improve dataset curation to overcome this problem. Unfortunately, these methods are generally ill-suited or even counterproductive in the network security domain, where they often result in unrealistic or poor-quality datasets. To address this issue, we propose a new closed-loop ML pipeline that leverages explainable ML tools to guide the network data collection in an iterative fashion. To ensure the data’s realism and quality, we require that the new datasets should be endogenously collected in this iterative process, thus advocating for a gradual removal of data-related problems to improve model generalizability. To realize this capability, we develop a data-collection platform, netUnicorn, that takes inspiration from the classic “hourglass” model and is implemented as its “thin waist" to simplify data collection for different learning problems from diverse network environments. The proposed system decouples data-collection intents from the deployment mechanisms and disaggregates these high-level intents into smaller reusable, self-contained tasks. We demonstrate how netUnicorn simplifies collecting data for different learning problems from multiple network environments and how the proposed iterative data collection improves a model’s generalizability
more »
« less
- PAR ID:
- 10544293
- Publisher / Repository:
- ACM
- Date Published:
- ISBN:
- 9798400700507
- Page Range / eLocation ID:
- 2217 to 2231
- Format(s):
- Medium: X
- Location:
- Copenhagen Denmark
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract This review paper examines the application and challenges of machine learning (ML) in intelligent welding processes within the automotive industry, focusing on resistance spot welding (RSW) and laser welding. RSW is predominant in body-in-white assembly, while laser welding is critical for electric vehicle battery packs due to its precision and compatibility with dissimilar materials. The paper categorizes ML applications into three key areas: sensing, in-process decision-making, and post-process optimization. It reviews supervised learning models for defect detection and weld quality prediction, unsupervised learning for feature extraction and data clustering, and emerging generalizable ML approaches like transfer learning and federated learning that enhance adaptability across different manufacturing conditions. Additionally, the paper highlights the limitations of current ML models, particularly regarding generalizability when moving from lab environments to real-world production, and discusses the importance of adaptive learning techniques to address dynamically changing conditions. Case studies like virtual sensing, defect detection in RSW, and optimization in laser welding illustrate practical applications. The paper concludes by identifying future research directions to improve ML adaptability and robustness in high-variability manufacturing environments, aiming to bridge the gap between experimental ML models and real-world implementation in automotive welding.more » « less
-
Several recent research efforts have proposed Machine Learning (ML)-based solutions that can detect complex patterns in network traffic for a wide range of network security problems. However, without understanding how these black-box models are making their decisions, network operators are reluctant to trust and deploy them in their production settings. One key reason for this reluctance is that these models are prone to the problem of underspecification, defined here as the failure to specify a model in adequate detail. Not unique to the network security domain, this problem manifests itself in ML models that exhibit unexpectedly poor behavior when deployed in real-world settings and has prompted growing interest in developing interpretable ML solutions (e.g., decision trees) for “explaining” to humans how a given black-box model makes its decisions. However, synthesizing such explainable models that capture a given black-box model’s decisions with high fidelity while also being practical (i.e., small enough in size for humans to comprehend) is challenging. In this paper, we focus on synthesizing high-fidelity and low-complexity decision trees to help network operators determine if their ML models suffer from the problem of underspecification. To this end, we present TRUSTEE, a framework that takes an existing ML model and training dataset generate a high-fidelity, easy-to-interpret decision tree, and associated trust report. Using published ML models that are fully reproducible, we show how practitioners can use TRUSTEE to identify three common instances of model underspecification, i.e., evidence of shortcut learning, spurious correlations, and vulnerability to out-of-distribution samples.more » « less
-
Goal, S (Ed.)Machine Learning models are widely utilized in a variety of applications, including Intelligent Transportation Systems (ITS). As these systems are operating in highly dynamic environments, they are exposed to numerous security threats that cause Data Quality (DQ) variations. Among such threats are network attacks that may cause data losses. We evaluate the influence of these factors on the image DQ and consequently on the image ML model performance. We propose and investigate Federated Learning (FL) as the way to enhance the overall level of privacy and security in ITS, as well as to improve ML model robustness to possible DQ variations in real-world applications. Our empirical study conducted with traffic sign images and YOLO, VGG16 and ResNet models proved the greater robustness of FL-based architecture over a centralized one.more » « less
-
Abstract BackgroundMeasuring parathyroid hormone-related peptide (PTHrP) helps diagnose the humoral hypercalcemia of malignancy, but is often ordered for patients with low pretest probability, resulting in poor test utilization. Manual review of results to identify inappropriate PTHrP orders is a cumbersome process. MethodsUsing a dataset of 1330 patients from a single institute, we developed a machine learning (ML) model to predict abnormal PTHrP results. We then evaluated the performance of the model on two external datasets. Different strategies (model transporting, retraining, rebuilding, and fine-tuning) were investigated to improve model generalizability. Maximum mean discrepancy (MMD) was adopted to quantify the shift of data distributions across different datasets. ResultsThe model achieved an area under the receiver operating characteristic curve (AUROC) of 0.936, and a specificity of 0.842 at 0.900 sensitivity in the development cohort. Directly transporting this model to two external datasets resulted in a deterioration of AUROC to 0.838 and 0.737, with the latter having a larger MMD corresponding to a greater data shift compared to the original dataset. Model rebuilding using site-specific data improved AUROC to 0.891 and 0.837 on the two sites, respectively. When external data is insufficient for retraining, a fine-tuning strategy also improved model utility. ConclusionsML offers promise to improve PTHrP test utilization while relieving the burden of manual review. Transporting a ready-made model to external datasets may lead to performance deterioration due to data distribution shift. Model retraining or rebuilding could improve generalizability when there are enough data, and model fine-tuning may be favorable when site-specific data is limited.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3576915.3623075
