skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Model Validation Based on Value-of-Information Theory
Abstract The modeling and simulation community has devoted considerable attention to the question of model validity. Their work has focused on the concept of “accuracy,” loosely defined as the difference between a model-computed result and a real-world result. This paper makes use of an example case that results in a paradox to illustrate weaknesses in an accuracy-focused approach, and proposes in its stead a value-focused approach based on classical decision theory. Instead of advocating the use of a model based on its accuracy, this work advocates using a model if it adds value to the overall application thus relating validation directly to system performance. The approach fills significant gaps in the current theory, notably providing a clearly defined validity metric and a fundamental rationale for the use of this metric.  more » « less
Award ID(s):
1923164
PAR ID:
10530966
Author(s) / Creator(s):
Publisher / Repository:
American Society of Mechanical Engineers
Date Published:
ISBN:
978-0-7918-8734-9
Format(s):
Medium: X
Location:
Boston, Massachusetts, USA
Sponsoring Org:
National Science Foundation
More Like this
  1. ; (, Water Resources Research)
    Abstract Environmental decisions with substantial social and environmental implications are regularly informed by model predictions, incurring inevitable uncertainty. The selection of a set of model predictions to inform a decision is usually based on model performance, measured by goodness‐of‐fit metrics. Yet goodness‐of‐fit metrics have a questionable relationship to a model's value to end users, particularly when validation data are themselves uncertain. For example, decisions based on flow frequency models are not necessarily improved by adopting models with the best overall goodness of fit. We propose an alternative model evaluation approach based on the conditional value of sample information, first defined in 1961, which has found extensive use in sampling design optimization but which has not previously been used for model evaluation. The metric uses observations from a validation set to estimate the expected monetary costs associated with model prediction uncertainties. A model is only considered superior to alternatives if (i) its predictions reduce these costs and (ii) sufficient validation data are available to distinguish its performance from alternative models. By describing prediction uncertainties in monetary terms, the metric facilitates the communication of prediction uncertainty by end users, supporting the inclusion of uncertainty analysis in decision making. 
    more » « less
  2. (, 4th Annual Conf. on Computational Science & Computational Intelligence (CSCI'17); Dec 14-16, 2017; Las Vegas, Nevada, USA)
    The goal of this research is to investigate the feasibility of Structure Equation Modeling approach for developing a quantitative behavior model grounded on the Theory of Planned Behavior. Data collected from an IRB sanctioned pilot consisting of approximately 500 participants were used to develop the model. The validity of the model is evaluated based on Chi-square, p-value, and RMSEA for statistical power and goodness of fit. The utility of the model is studied through correlation analysis related to user engagement for self-health management. Example association pattern discovered from the analysis was illustrated for its use in digital health software development. 
    more » « less
  3. ; (, ICCAD '20: Proceedings of the 39th International Conference on Computer-Aided Design)
    null (Ed.)
    Understanding accuracy and the tradeoffs it entails is key to evaluating the growing list of stochastic computing (SC) circuit designs. Due to shortcomings of current SC error theory, simulation has become the standard way to estimate a circuit's accuracy. However, simulation can demand large computational resources and lead to uncertain, misleading, or unexplainable results. A soundly based analytic approach is therefore preferable to simulation. In this work, we first show the input value distribution's large influence on circuit accuracy. Then we develop a Bayesian error analysis methodology which uses the input value distribution as a prior to inform better accuracy estimates. This error formulation introduces concepts new to SC such as estimator dominance and points to ways of improving simulation-based accuracy estimates. Orthogonal to the Bayesian ideas, we also show how to use bias-variance decomposition to simplify and aggregate the effects of SC's many error sources. We present techniques that use the beta distribution to model the stochastic number value distribution. Finally, we demonstrate the use of these ideas to improve the accuracy and analysis of an SC-based neural network. 
    more » « less
  4. ; ; ; ; ; ; (, La Matematica)
    Abstract Graphs in metric spaces appear in a wide range of data sets, and there is a large body of work focused on comparing, matching, or analyzing collections of graphs in different ambient spaces. In this survey, we provide an overview of a diverse collection of distance measures that can be defined on the set of finite graphs immersed (and in some cases, embedded) in a metric space. For each of the distance measures, we recall their definitions and investigate which of the properties of a metric they satisfy. Furthermore we compare the distance measures based on these properties and discuss their computational complexity. 
    more » « less
  5. ; (, Discrete and Continuous Dynamical Systems - S)
    We introduce a new approach to the study of modulation of high-frequency periodic wave patterns, based on pseudodifferential analysis, multi-scale expansion, and Kreiss symmetrizer estimates like those in hyperbolic and hyperbolic-parabolic boundary-value theory. Key ingredients are local Floquet transformation as a preconditioner removing large derivatives in the normal direction of background rapidly oscillating fronts and the use of the periodic Evans function of Gardner to connect spectral information on component periodic waves to block structure of the resulting approximately constant-coefficient resolvent ODEs. Our main result is bounded-time existence and validity to all orders of large-amplitude smooth modulations of planar periodic solutions of multi-D reaction diffusion systems in the high-frequency/small wavelength limit. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email