Search for: All records

This article studies the adaptive optimal stationary control of continuous-time linear stochastic systems with both additive and multiplicative noises, using reinforcement learning techniques. Based on policy iteration, a novel off-policy reinforcement learning algorithm, named optimistic least-squares-based policy iteration, is proposed, which is able to find iteratively near-optimal policies of the adaptive optimal stationary control problem directly from input/state data without explicitly identifying any system matrices, starting from an initial admissible control policy. The solutions given by the proposed optimistic least-squares-based policy iteration are proved to converge to a small neighborhood of the optimal solution with probability one, under mild conditions. The application of the proposed algorithm to a triple inverted pendulum example validates its feasibility and effectiveness. 

Demand dispatch is the science of extracting virtual energy storage through the automatic control of deferrable loads to provide balancing or regulation services to the grid, while maintaining consumer-end quality of service.The control of a large collection of heterogeneous loads is in part a resource allocation problem, since different classes of loads are more valuable for different services. The goal of this paper is to unveil the structure of the optimal solution to the resource allocation problem, and investigate short-term market implications. It is found that the marginal cost for each load class evolves in a two-dimensional subspace: spanned by a co-state process and its derivative. The resource allocation problem is recast to construct a dynamic competitive equilibrium model, in which the consumer utility is the negative of the cost of deviation from ideal QoS. It is found that a competitive equilibrium exists with the equilibrium price equal to the negative of an optimal co-state process. Moreover, the equilibrium price is different than what would be obtained based on the standard assumption that the consumer's utility is a function of power consumption.