skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: Measures of Model Performance Based On the Log Accuracy Ratio
Abstract Quantitative assessment of modeling and forecasting of continuous quantities uses a variety of approaches. We review existing literature describing metrics for forecast accuracy and bias, concentrating on those based on relative errors and percentage errors. Of these accuracy metrics, the mean absolute percentage error (MAPE) is one of the most common across many fields and has been widely applied in recent space science literature and we highlight the benefits and drawbacks of MAPE and proposed alternatives. We then introduce the log accuracy ratio and derive from it two metrics: the median symmetric accuracy and the symmetric signed percentage bias. Robust methods for estimating the spread of a multiplicative linear model using the log accuracy ratio are also presented. The developed metrics are shown to be easy to interpret, robust, and to mitigate the key drawbacks of their more widely used counterparts based on relative errors and percentage errors. Their use is illustrated with radiation belt electron flux modeling examples.  more » « less
Award ID(s):
1663770
PAR ID:
10480331
Author(s) / Creator(s):
; ;
Publisher / Repository:
Wiley
Date Published:
Journal Name:
Space Weather
Volume:
16
Issue:
1
ISSN:
1542-7390
Page Range / eLocation ID:
69 to 88
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. IntroductionThe objective of this study is to develop predictive models for rocking-induced permanent settlement in shallow foundations during earthquake loading using stacking, bagging and boosting ensemble machine learning (ML) and artificial neural network (ANN) models. MethodsThe ML models are developed using supervised learning technique and results obtained from rocking foundation experiments conducted on shaking tables and centrifuges. The overall performance of ML models are evaluated using k-fold cross validation tests and mean absolute percentage error (MAPE) and mean absolute error (MAE) in their predictions. ResultsThe performances of all six nonlinear ML models developed in this study are relatively consistent in terms of prediction accuracy with their average MAPE varying between 0.64 and 0.86 in final k-fold cross validation tests. DiscussionThe overall average MAE in predictions of all nonlinear ML models are smaller than 0.006, implying that the ML models developed in this study have the potential to predict permanent settlement of rocking foundations with reasonable accuracy in practical applications. 
    more » « less
  2. The objective of this study is to develop data-driven predictive models for peak rotation and factor of safety for tipping-over failure of rocking shallow foundations during earthquake loading using multiple nonlinear machine learning (ML) algorithms and a supervised learning technique. Centrifuge and shaking table experimental results on rocking foundations have been used for the development of k-nearest neighbors regression (KNN), support vector regression (SVR), and random forest regression (RFR) models. The input features to ML models include critical contact area ratio of foundation; slenderness ratio and rocking coefficient of rocking system; peak ground acceleration and Arias intensity of earthquake motion; and a categorical binary feature that separates sandy soil foundations from clayey soil foundations. Based on repeated k-fold cross validation tests of models, we found that the overall average mean absolute percentage errors (MAPE) in predictions of all three nonlinear ML models varied between 0.46 and 0.60, outperforming a baseline multivariate linear regression ML model with corresponding MAPE of 0.68 to 0.75. The input feature importance analysis reveals that the peak rotation and tipping-over stability of rocking foundations are more sensitive to ground motion demand parameters than to rocking foundation capacity parameters or type of soil. 
    more » « less
  3. Abstract Machine learning interatomic potentials (MLIPs) are a promising technique for atomic modeling. While small errors are widely reported for MLIPs, an open concern is whether MLIPs can accurately reproduce atomistic dynamics and related physical properties in molecular dynamics (MD) simulations. In this study, we examine the state-of-the-art MLIPs and uncover several discrepancies related to atom dynamics, defects, and rare events (REs), compared to ab initio methods. We find that low averaged errors by current MLIP testing are insufficient, and develop quantitative metrics that better indicate the accurate prediction of atomic dynamics by MLIPs. The MLIPs optimized by the RE-based evaluation metrics are demonstrated to have improved prediction in multiple properties. The identified errors, the evaluation metrics, and the proposed process of developing such metrics are general to MLIPs, thus providing valuable guidance for future testing and improvements of accurate and reliable MLIPs for atomistic modeling. 
    more » « less
  4. Abstract We provide a new algorithm for mass‐balance calculations in petrology and geochemistry based on the log‐ratio approach championed initially by John Aitchison (e.g., Aitchison, 1982,https://doi.org/10.1111/j.2517-6161.1982.tb01195.x; Aitchison, 1984,https://doi.org/10.1007/bf01029316) along with the underlying principles, mathematical frameworks, and data requirements. Log‐ratio Inversion of Mixed End‐members (LIME) is written in MATLAB and calculates phase proportions in an experiment or rock given a bulk composition, the composition of each phase, and the associated compositional uncertainties. An important advantage of LIME is that performing the mass‐balance calculation in inverse log‐ratio space constrains phase proportions to be between 0 and 100 wt.%. Further, the resulting LIME phase proportions provide realistic estimates of uncertainty regardless of data distribution. These two characteristics of LIME improve upon standard multiple linear regression techniques, which may yield negative values for phase proportions if non‐constrained or report oversimplified symmetric errors. Primary applications of LIME include estimating phase abundances, calculating melting and metamorphic reaction stoichiometries, and checking for open system behavior in phase equilibria experiments. The technique presented here covers whole‐rock analysis, mineralogy, and phase abundance, but could be extended to isotopic tracers, trace element modeling, and regolith component un‐mixing. We highlight the importance of uncertainty estimations for phase abundances to the fields of petrology and geochemistry by comparing our results from LIME to previously published mass‐balance calculations. Furthermore, we present case studies that demonstrate the role of mass‐balance calculations in determining magma crystallinity and defining melting reactions. 
    more » « less
  5. Electric load forecasting refers to forecasting the electricity demand at aggregated levels. Utilities use the predictions of this technique to keep a balance between electricity generation and consumption at each time and make accurate decision for power system planning, operations, and maintenance, etc. Based on prediction time horizon, electric load forecasting is classified to very short-term, short-term, medium-term, and long-term. In this paper, a multiple output Gaussian processes with multiple kernel learning is proposed to predict short-term electric load forecasting (predicting 24 load values for the next day) based on load, temperature, and dew point values of previous days. Mean absolute percentage error (MAPE) is used as a measure of prediction accuracy. By comparing MAPE values of the proposed method with the persistence method, it can been seen that the proposed method improves the persistence method MAPE up to 4%. 
    more » « less