skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: A Framework for the Dynamic Programming Principle and Martingale-Generated Control Correspondences
We construct an abstract framework in which the dynamic programming principle (DPP) can be readily proven. It encompasses a broad range of common stochastic control problems in the weak formulation, and deals with problems in the “martingale formulation” with particular ease. We give two illustrations; first, we establish the DPP for general controlled diffusions and show that their value functions are viscosity solutions of the associated Hamilton–Jacobi–Bellman equations under minimal conditions. After that, we show how to treat singular control on the example of the classical monotone-follower problem.  more » « less
Award ID(s):
1815017
PAR ID:
10108978
Author(s) / Creator(s):
;
Date Published:
Journal Name:
Applied Mathematics & Optimization
ISSN:
0095-4616
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. We study a stochastic control/stopping problem with a series of inequality-type and equality-type expectation constraints in a general non-Markovian framework. We demonstrate that the stochastic control/stopping problem with expectation constraints (CSEC) is independent of a specific probability setting and is equivalent to the constrained stochastic control/stopping problem in weak formulation (an optimization over joint laws of Brownian motion, state dynamics, diffusion controls and stopping rules on an enlarged canonical space). Using a martingale-problem formulation of controlled SDEs in spirit of Stroock and Varadhan (2006), we characterize the probability classes in weak formulation by countably many actions of canonical processes, and thus obtain the upper semi-analyticity of the CSEC value function. Then we employ a measurable selection argument to establish a dynamic programming principle (DPP) in weak formulation for the CSEC value function, in which the conditional expected costs act as additional states for constraint levels at the intermediate horizon. This article extends (El Karoui and Tan, 2013) to the expectation-constraint case. We extend our previous work (Bayraktar and Yao, 2024) to the more complicated setting where the diffusion is controlled. Compared to that paper the topological properties of diffusion-control spaces and the corresponding measurability are more technically involved which complicate the arguments especially for the measurable selection for the super-solution side of DPP in the weak formulation. 
    more » « less
  2. We study Mean Field stochastic control problems where the cost function and the state dynamics depend upon the joint distribution of the controlled state and the control process. We prove suitable versions of the Pontryagin stochastic maximum principle, both in necessary and in sufficient form, which extend the known conditions to this general framework. Furthermore, we suggest a variational approach to study a weak formulation of these control problems. We show a natural connection between this weak formulation and optimal transport on path space, which inspires a novel discretization scheme. 
    more » « less
  3. null (Ed.)
    The significance of multiple number of donor-acceptor entities on a central electron donor in a star-shaped molecular system in improving light energy harvesting ability is reported. For this, donor-acceptor-donor type conjugates comprised up to three entities ferrocenyl (Fc)-diketopyrrolopyrrole (DPP) onto a central triphenylamine (TPA), (4-6) by the Pd-catalyzed Sonogashira cross–coupling reactions have been newly synthesized and characterized. Donor-acceptor conjugates possessing diketopyrrolopyrrole (1 to 3 entities) onto the central triphenylamine, (1-3) served as reference dyads while monomeric DPP and Fc-DPP served as control compounds. Both DPP and Fc-DPP carrying conjugates exhibited red-shifted absorption compared to their respective control compounds revealing existence of ground state interactions. Furthermore, DPP fluorescence in 4-6 was found to be quantitatively quenched while for 1-3, this property varied between 73-65% suggesting occurrence moderate amounts of excited state events. The electrochemical investigations exhibited an additional low potential oxidation in the case of Fc-DPP-TPA based derivatives (4-6) owing to the presence of ferrocene unit(s). This was in addition to DPP and TPA redox peaks. Using spectral, electrochemical and computational studies, Gibbs free-energy calculations were performed to visualize excited state charge separation (GCS) in these donor-acceptor conjugates as a function of different number of Fc-DPP entities. Formation of Fc+-DPP•--TPA charge separated states (CSS) in the case of 4-6 was evident. Using spectroelectrochemical studies, spectrum of CSS was deduced. Finally, femtosecond transient absorption spectral studies were performed to gather information on excited state charge separation. Increasing the number of Fc-DPP entities in 4-6 improved charge separation rates. Surprisingly, lifetime of the charge separated state, Fc+-DPP•--TPA was found to persist longer with an increase in the number of Fc-DPP entities in 4-6 as compared to Fc-DPP-control and simple DPP derived donor-acceptor conjugates in literature. This unprecedented result has been attributed to subtle changes in GCS and GCR and the associated electron coupling between different entities. 
    more » « less
  4. The temporal dynamics of morphogen presentation impacts transcriptional responses and tissue patterning (1). However, the mechanisms controlling morphogen release are far from clear. We found that inwardly rectifying potassium (Irk) channels regulate endogenous transient increases in intracellular calcium and Bone Morphogenetic Protein (BMP/Dpp) release for Drosophila wing development (2). Inhibition of Irk channels reduces BMP/Dpp signaling, and ultimately disrupts wing morphology (2, 3). Ion channels impact development of several tissues and organisms in which BMP signaling is essential (2-15). In neurons and pancreatic beta cells, Irk channels modulate membrane potential to affect intracellular Ca++ to control secretion of neurotransmitters and insulin (15-21). Based on Irk activity in neurons, we hypothesized that electrical activity controls endoplasmic reticulum Ca++ release into the cytoplasm to regulate the release of BMP. To test this hypothesis, we reduced expression of proteins that control endoplasmic reticulum calcium (Stim, Orai, SERCA, SK, and Best2) and documented wing phenotypes. We found that reduced Stim and SERCA function decreases amplitude and frequency of endogenous calcium transients in the wing disc and reduced Dpp/BMP release in the wing disc. Together, our results suggest control of endoplasmic reticulum is required for Dpp/BMP release. 
    more » « less
  5. We aim to preserve a large amount of data generated insidebase station-less sensor networks(BSNs) while considering that sensor nodes are selfish. BSNs refer to emerging sensing applications deployed in challenging and inhospitable environments (e.g., underwater exploration); as such, there do not exist data-collecting base stations in the BSN to collect the data. Consequently, the generated data has to be stored inside the BSN before uploading opportunities become available. Our goal is to preserve the data inside the BSN with minimum energy cost by incentivizing the storage- and energy-constrained sensor nodes to participate in the data preservation process. We refer to the problem as DPP:datapreservationproblem in the BSN. Previous research assumes that all the sensor nodes are cooperative and that sensors have infinite battery power and design a minimum-cost flow-based data preservation solution. However, in a distributed setting and under different control, the resource-constrained sensor nodes could behave selfishly only to conserve their resources and maximize their benefit. In this article, we first solve DPP by designing an integer linear programming (ILP)-based optimal solution without considering selfishness. We then establish a game-theoretical framework that achieves provably truthful and optimal data preservation in BSNs. For a special case of DPP wherein nodes are not energy-constrained, referred to as DPP-W, we design a data preservation game DPG-1 that integrates algorithmic mechanism design (AMD) and a more efficient minimum cost flow-based data preservation solution. We show that DPG-1 yields dominant strategies for sensor nodes and delivers truthful and optimal data preservation. For the general case of DPP (wherein nodes are energy-constrained), however, DPG-1 fails to achieve truthful and optimal data preservation. Utilizing packet-level flow observation of sensor node behaviors computed by minimum cost flow and ILP, we uncover the cause of the failure of the DPG-1. It is due to the packet dropping by the selfish nodes that manipulate the AMD technique. We then design a data preservation game DPG-2 for DPP that traces and punishes manipulative nodes in the BSN. We show that DPG-2 delivers dominant strategies for truth-telling nodes and achieves provably optimal data preservation with cheat-proof guarantees. Via extensive simulations under different network parameters and dynamics, we show that our games achieve system-wide data preservation solutions with optimal energy cost while enforcing truth-telling of sensor nodes about their private cost types. One salient feature of our work is its integrated game theory and network flows approach. With the observation of flow level sensor node behaviors provided by the network flows, our proposed games can synthesize “microscopic” (i.e., selfish and local) behaviors of sensor nodes and yield targeted “macroscopic” (i.e., optimal and global) network performance of data preservation in the BSN. 
    more » « less