An official website of the United States government
The functions of an autonomous system can generally be partitioned into those concerned with perception and those concerned with action. Perception builds and maintains an internal model of the world (i.e., the system's environment) that is used to plan and execute actions to accomplish a goal established by human supervisors. Accordingly, assurance decomposes into two parts: a) ensuring that the model is an accurate representation of the world as it changes through time and b) ensuring that the actions are safe (and e ective), given the model. Both perception and action may employ AI, including machine learning (ML), and these present challenges to assurance. However, it is usually feasible to guard the actions with traditionally engineered and assured monitors, and thereby ensure safety, given the model. Thus, the model becomes the central focus for assurance. We propose an architecture and methods to ensure the accuracy of models derived from sensors whose interpretation uses AI and ML. Rather than derive the model from sensors bottom-up, we reverse the process and use the model to predict sensor interpretation. Small prediction errors indicate the world is evolving as expected and the model is updated accordingly. Large prediction errors indicate surprise, which may be due to errors in sensing or interpretation, or unexpected changes in the world (e.g., a pedestrian steps into the road). The former initiate error masking or recovery, while the latter requires revision to the model. Higher-level AI functions assist in diagnosis and execution of these tasks. Although this two-level architecture where the lower level does \predictive processing" and the upper performs more re ective tasks, both focused on maintenance of a world model, is derived by engineering considerations, it also matches a widely accepted theory of human cognition.
more »
« less
This content will become publicly available on August 26, 2026
Minimizing and quantifying uncertainty in AI-informed decisions: Applications in medicine
AI is now a cornerstone of modern dataset analysis. In many real world applications, practitioners are concerned with controlling specific kinds of errors, rather than minimizing the overall number of errors. For example, biomedical screening assays may primarily be concerned with mitigating the number of false positives rather than false negatives. Quantifying uncertainty in AI-based predictions, and in particular those controlling specific kinds of errors, remains theoretically and practically challenging. We develop a strategy called multidimensional informed generalized hypothesis testing (MIGHT) which we prove accurately quantifies uncertainty and confidence given sufficient data, and concomitantly controls for particular error types. Our key insight was that it is possible to integrate canonical cross-validation and parametric calibration procedures within a nonparametric ensemble method. Simulations demonstrate that while typical AI based-approaches cannot be trusted to obtain the truth, MIGHT can be. We apply MIGHT to answer an open question in liquid biopsies using circulating cell-free DNA (ccfDNA) in individuals with or without cancer: Which biomarkers, or combinations thereof, can we trust? Performance estimates produced by MIGHT on ccfDNA data have coefficients of variation that are often orders of magnitude lower than other state of the art algorithms such as support vector machines, random forests, and Transformers, while often also achieving higher sensitivity. We find that combinations of variable sets often decrease rather than increase sensitivity over the optimal single variable set because some variable sets add more noise than signal. This work demonstrates the importance of quantifying uncertainty and confidence—with theoretical guarantees—for the interpretation of real-world data.
more »
« less
- PAR ID:
- 10636371
- Author(s) / Creator(s):
- ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; more »
- Publisher / Repository:
- National Academy of Sciences
- Date Published:
- Journal Name:
- Proceedings of the National Academy of Sciences
- Volume:
- 122
- Issue:
- 34
- ISSN:
- 0027-8424
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Artificial intelligence (AI) and machine learning models are being increasingly deployed in real-world applications. In many of these applications, there is strong motivation to develop hybrid systems in which humans and AI algorithms can work together, leveraging their complementary strengths and weaknesses. We develop a Bayesian framework for combining the predictions and different types of confidence scores from humans and machines. The framework allows us to investigate the factors that influence complementarity, where a hybrid combination of human and machine predictions leads to better performance than combinations of human or machine predictions alone. We apply this framework to a large-scale dataset where humans and a variety of convolutional neural networks perform the same challenging image classification task. We show empirically and theoretically that complementarity can be achieved even if the human and machine classifiers perform at different accuracy levels as long as these accuracy differences fall within a bound determined by the latent correlation between human and machine classifier confidence scores. In addition, we demonstrate that hybrid human–machine performance can be improved by differentiating between the errors that humans and machine classifiers make across different class labels. Finally, our results show that eliciting and including human confidence ratings improve hybrid performance in the Bayesian combination model. Our approach is applicable to a wide variety of classification problems involving human and machine algorithms.more » « less
-
Cook, S.; Infante, N. (Ed.)In this theoretical paper, our aim is to start a conversation about how "levels" in mathematics are operationalized and defined, with a specific focus on "college level." We approach this from a lens of developmental stages, using this to propose an initial framework for describing how learners might progress along a developmental continuum delineated by the kinds of reasoning/justification, generalization/abstraction, and types of conceptions that they hold, rather than by the particular computations learners are able to do, or the kinds of mathematical objects with which learners are engaging.more » « less
-
Recent advances in the study of artificial and biological neural networks support the power of dynamic representations-computing with information stored as nontrivial limit-sets rather than fixed-point attractors. Understanding and manipulating these computations in nonlinear networks requires a theory of control for abstract objective functions. Towards this end, we consider two properties of limit-sets: their topological dimension and orientation (covariance) in phase space and combine these abstract properties into a single well-defined objective: conic control-invariant sets in the derivative space (i.e., the vector field). Real-world applications, such as neural-medicine, constrain which control laws are feasible with less-invasive controllers being preferable. To this end, we derive a feedback control-law for conic invariance which corresponds to constrained restructuring of the network connections as might occur with pharmacological intervention (as opposed to a physically separate control unit). We demonstrate the ease and efficacy of the technique in controlling the orientation and dimension of limit sets in high-dimensional neural networks.more » « less
-
Natural kinds are widely understood to be the real classifications of things that actually exist in the world. Natural kinds are the categories we tend to aim for when we seek to understand the world, as it really is. Discovering what these real classifications are is often considered to be the project of scientific research in many fields from astronomy and agronomy to zoology and zymurgy. When we discover something unfamiliar to us and we want to know what sort of thing it is, we might ask: ‘what kind is that?’ For instance, in a physics class, we might ask: ‘what kind of quark is that?’, where the answer might be: ‘that is a charm quark’. In biology, we might ask: ‘what kind of plant is that?’, with the answer being: ‘it is a Venus flytrap (Dionaea muscipula)’. Or, in chemistry, we might ask: ‘what element is that?’, with the answer being: ‘lithium’. Knowing that the thing we asked about is a member of that particular kind tells us a lot about it if it is a natural kind. Membership in a natural kind tells us that the thing in question shares many important characteristics with other things that are in the same natural kind category. For example, consider the category of Venus flytraps. All plants that belong to that category share many important characteristics; among these include: perennial flowering, carnivorous eating habits, capable of thigmonastic responses (closing their ‘trap’ when prey alight on their trichomes), ability to photosynthesise, belonging to the family Droseraceae and the kingdom Plantae. Some of these important characteristics and properties have been referred to as ‘essential’ by philosophers because they are the properties that are thought to be necessary for the thing to be a member of that natural kind. That means that if the thing does not have those necessary properties, then it cannot be a member of that natural kind. When we ask the question: ‘what kind is that?’, we do not always discover natural kinds. Sometimes when we ask: ‘what kind of thing is that?’, we find out, for instance, that this thing that we are asking about is green. This means we find out that it belongs to the category of green things. The category of green things is a kind category, but it is not a natural kind. The category of all green things includes the Venus flytrap but also green traffic lights, green tea, guacamole, collards, and dark jade-painted 1978 Ford Mustang sportscars. What all of these things have in common is that they are all green. However, they do not share any other properties or characteristics apart from being green. Green kinds of things are not natural kinds like those mentioned earlier. The kinds that are picked out by the classifications of charm quark, Venus flytrap, and lithium are considered to be very different from the classification of green things. Whilst all charm quarks, all Venus flytraps, and all samples of lithium are each considered to be classifications of natural kinds, the category of green things is not. The philosophical question that arises is: ‘what makes classifications like that of the natural kind that includes all charm quarks natural and classifications like that of all green things not natural?’ Put a different way: ‘what makes something a natural kind and how can we tell the difference between natural kinds and what we might call “artificial kinds”, like the grouping of green things?’ A popular answer to this question is that natural kinds pick out natural groupings whose existence in the world is not dependent upon human interests or activities, whereas artificial kinds pick out groupings whose existence in the world is dependent upon human interests or activities. However, others have provided substantial evidence challenging this claim, arguing that there are at least some natural kinds that are dependent upon human activities and practices for their existence. In addition to questions concerning what qualifies as naturalness in natural kinds and what is the distinction between natural and artificial kinds, philosophical discussion also focuses on the metaphysics of natural kinds and the epistemic value of natural kinds. A perennial question widely debated is whether the classifications used in scientific disciplines – physics, chemistry, biology, neuroscience, geology, linguistics, anthropology, and more –really do map on to a natural classification that really exists in the world. That is, are the ways we partition elements in chemistry, organisms in biology, or quarks in physics, the same partitionings that naturally exist? A lot of the literature on natural kinds relies on using examples that are thought to be quintessential natural kinds, like biological species and chemical elements. But others argue that there is clear evidence that many, if not most, biological species and chemical elements are not natural kinds, especially if membership within a natural kind requires possession of an essential property. Within the discussion of natural kinds, there are also questions with regard to the conditions of membership that challenge the view that natural kinds membership is determined by the possession of a particular essence. Instead of the possession of a particular essence, some argue that membership in a natural kind may instead be determined by the possession of a cluster of properties, a relationship, or something else. In many of these discussions, Plato’s metaphor of carving nature at its joints is used to describe the mapping of natural classifications onto natural kinds by the implied comparison to the butchering of an animal along its natural divisions (knuckles, limbs, etc.) rather than partitioning it in a way that does not coincide with the animal’s body structure. Whilst the metaphor helps explain the nature of natural kinds, it does so by assuming nature is that which is pre-partitioned.more » « less
