More Like this

1. The contribution of distributional activities of dopamine in the exploration

   Wang, Mien Brabeeba; Lynch, Nancy; Halassa, Michael (June 2023, 9th workshop on Biological Distributed Algorithms (BDA))

   Decision making in natural settings requires efficient exploration to handle uncertainty. Since associations between actions and outcomes are uncertain, animals need to balance the explorations and exploitation to select the actions that lead to maximal rewards. The computa- tional principles by which animal brains explore during decision-making are poorly understood. Our challenge here was to build a biologically plausible neural network that efficiently explores an environment and understands its effectiveness mathematically. One of the most evolutionarily conserved and important systems in decision making is basal ganglia (BG)1. In particular, the dopamine activities (DA) in BG is thought to represent reward prediction error (RPE) to facilitate reinforcement learning2. Therefore, our starting point is a cortico-BG loop motif3. This network adjusts exploration based on neuronal noises and updates its value estimate through RPE. To account for the fact that animals adjust exploration based on experience, we modified the network in two ways. First, it is recently discovered that DA does not simply represent the scalar RPE value; rather it represents RPE in a distribution4. We incorporated the distributional RPE framework and further the hypothesis, allowing an RPE distribution to update the posterior of action values encoded by cortico-BG connections. Second, it is known that the firing in the layer 2/3 of cortex fires is variable and sparse5. Our network thus included a random sparsification of cortical activity as a mechanism of sampling from this posterior for experience-based exploration. Combining these two features, our network is able to take the uncertainty of our value estimates into account to accomplish efficient exploration in a variety of environments. 

   more » « less
2. Theory and model of thalamocortical processing in decision making under uncertainty. Thalamocortical Interactions Gordon Research Conference 2024, Ventura, California, February 2024. Abstract, Poster.

   Wang, Mien Brabeeba; Lynch, Nancy; Halassa, Michael (February 2024, Thalamocortical Interactions Gordon Research Conference 2024)

   Animals flexibly select actions that maximize future rewards despite facing uncertainty in sen- sory inputs, action-outcome associations or contexts. The computational and circuit mechanisms underlying this ability are poorly understood. A clue to such computations can be found in the neural systems involved in representing sensory features, sensorimotor-outcome associations and contexts. Specifically, the basal ganglia (BG) have been implicated in forming sensorimotor-outcome association [1] while the thalamocortical loop between the prefrontal cortex (PFC) and mediodorsal thalamus (MD) has been shown to engage in contextual representations [2, 3]. Interestingly, both human and non-human animal experiments indicate that the MD represents different forms of uncertainty [3, 4]. However, finding evidence for uncertainty representation gives little insight into how it is utilized to drive behavior. Normative theories have excelled at providing such computational insights. For example, de- ploying traditional machine learning algorithms to fit human decision-making behavior has clarified how associative uncertainty alters exploratory behavior [5, 6]. However, despite their computa- tional insight and ability to fit behaviors, normative models cannot be directly related to neural mechanisms. Therefore, a critical gap exists between what we know about the neural representa- tion of uncertainty on one end and the computational functions uncertainty serves in cognition. This gap can be filled with mechanistic neural models that can approximate normative models as well as generate experimentally observed neural representations. In this work, we build a mechanistic cortico-thalamo-BG loop network model that directly fills this gap. The model includes computationally-relevant mechanistic details of both BG and thalamocortical circuits such as distributional activities of dopamine [7] and thalamocortical pro- jection modulating cortical effective connectivity [3] and plasticity [8] via interneurons. We show that our network can more efficiently and flexibly explore various environments compared to com- monly used machine learning algorithms and we show that the mechanistic features we include are crucial for handling different types of uncertainty in decision-making. Furthermore, through derivation and mathematical proofs, we approximate our models to two novel normative theories. We show mathematically the first has near-optimal performance on bandit tasks. The second is a generalization on the well-known CUMSUM algorithm, which is known to be optimal on single change point detection tasks [9]. Our normative model expands on this by detecting multiple sequential contextual changes. To our knowledge, our work is the first to link computational in- sights, normative models and neural realization together in decision-making under various forms of uncertainty. 

   more » « less
3. Solving Markov decision processes for network-level post-hazard recovery via simulation optimization and rollout

   Sarkale, Y.; Nozhati, S.; Chong, E. K.; Ellingwood, B. R.; Mahmoud, H. (August 2019, 2018 IEEE 14th International Conference on Automation Science and Engineering (CASE))

   Computation of optimal recovery decisions for community resilience assurance post-hazard is a combinatorial decision-making problem under uncertainty. It involves solving a large-scale optimization problem, which is significantly aggravated by the introduction of uncertainty. In this paper, we draw upon established tools from multiple research communities to provide an effective solution to this challenging problem. We provide a stochastic model of damage to the water network (WN) within a testbed community following a severe earthquake and compute near-optimal recovery actions for restoration of the water network. We formulate this stochastic decision-making problem as a Markov Decision Process (MDP), and solve it using a popular class of heuristic algorithms known as rollout. A simulation-based representation of MDPs is utilized in conjunction with rollout and the Optimal Computing Budget Allocation (OCBA) algorithm to address the resulting stochastic simulation optimization problem. Our method employs non-myopic planning with efficient use of simulation budget. We show, through simulation results, that rollout fused with OCBA performs competitively with respect to rollout with total equal allocation (TEA) at a meagre simulation budget of 5-10% of rollout with TEA, which is a crucial step towards addressing large-scale community recovery problems following natural disasters. 

   more » « less
4. Optimal Joint Defense and Monitoring for Networks Security under Uncertainty: A POMDP‐Based Approach

   https://doi.org/10.1049/2024/7966713  

   Kazeminajafabadi, Armita; Imani, Mahdi; Bakhshi, ed., Taimur (May 2024, IET Information Security)

   The increasing interconnectivity in our infrastructure poses a significant security challenge, with external threats having the potential to penetrate and propagate throughout the network. Bayesian attack graphs have proven to be effective in capturing the propagation of attacks in complex interconnected networks. However, most existing security approaches fail to systematically account for the limitation of resources and uncertainty arising from the complexity of attacks and possible undetected compromises. To address these challenges, this paper proposes a partially observable Markov decision process (POMDP) model for network security under uncertainty. The POMDP model accounts for uncertainty in monitoring and defense processes, as well as the probabilistic attack propagation. This paper develops two security policies based on the optimal stationary defense policy for the underlying POMDP state process (i.e., a network with known compromises): the estimation‐based policy that performs the defense actions corresponding to the optimal minimum mean square error state estimation and the distribution‐based policy that utilizes the posterior distribution of network compromises to make defense decisions. Optimal monitoring policies are designed to specifically support each of the defense policies, allowing dynamic allocation of monitoring resources to capture network vulnerabilities/compromises. The performance of the proposed policies is examined in terms of robustness, accuracy, and uncertainty using various numerical experiments. 

   more » « less
5. Factors influencing wildfire management decisions after the 2009 US federal policy update

   https://doi.org/10.1071/WF23129  

   Fillmore, Stephen D.; McCaffrey, Sarah; Bean, Rachel; Evans, Alexander M.; Iniguez, Jose; Thode, Andrea; Smith, Alistair M.; Thompson, Matthew P. (January 2024, International Journal of Wildland Fire)

   Background The decision making process undertaken during wildfire responses is complex and prone to uncertainty. In the US, decisions federal land managers make are influenced by numerous and often competing factors. Aims To assess and validate the presence of decision factors relevant to the wildfire decision making context that were previously known and to identify those that have emerged since the US federal wildfire policy was updated in 2009. Methods Interviews were conducted across the US while wildfires were actively burning to elucidate time-of-fire decision factors. Data were coded and thematically analysed. Key results Most previously known decision factors as well as numerous emergent factors were identified. Conclusions To contextualise decision factors within the decision making process, we offer a Wildfire Decision Framework that has value for policy makers seeking to improve decision making, managers improving their process and wildfire social science researchers. Implications Managers may gain a better understanding of their decision environment and use our framework as a tool to validate their deliberations. Researchers may use these data to help explain the various pressures and influences modern land and wildfire managers experience. Policy makers and agencies may take institutional steps to align the actions of their staff with desired wildfire outcomes. 

   more » « less