Driven by recent successes in twoplayer, zerosum game solving and playing, artificial intelligence work on games has increasingly focused on algorithms that produce equilibriumbased strategies. However, this approach has been less effective at producing competent players in generalsum games or those with more than two players than in twoplayer, zerosum games. An appealing alternative is to consider adaptive algorithms that ensure strong performance in hindsight relative to what could have been achieved with modified behavior. This approach also leads to a gametheoretic analysis, but in the correlated play that arises from joint learning dynamics rather than factored agent behavior at equilibrium. We develop and advocate for this hindsight rationality framing of learning in general sequential decisionmaking settings. To this end, we reexamine mediated equilibrium and deviation types in extensiveform games, thereby gaining a more complete understanding and resolving past misconceptions. We present a set of examples illustrating the distinct strengths and weaknesses of each type of equilibrium in the literature, and prove that no tractable concept subsumes all others. This line of inquiry culminates in the definition of the deviation and equilibrium classes that correspond to algorithms in the counterfactual regret minimization (CFR) family, relating them to all others inmore »
Efficient Deviation Types and Learning for Hindsight Rationality in ExtensiveForm Games
Hindsight rationality is an approach to playing generalsum games that prescribes noregret learning dynamics for individual agents with respect to a set of deviations, and further describes jointly rational behavior among multiple agents with mediated equilibria. To develop hindsight rational learning in sequential decisionmaking settings, we formalize behavioral deviations as a general class of deviations that respect the structure of extensiveform games. Integrating the idea of time selection into counterfactual regret minimization (CFR), we introduce the extensiveform regret minimization (EFR) algorithm that achieves hindsight rationality for any given set of behavioral deviations with computation that scales closely with the complexity of the set. We identify behavioral deviation subsets, the partial sequence deviation types, that subsume previously studied types and lead to efficient EFR instances in games with moderate lengths. In addition, we present a thorough empirical analysis of EFR instantiated with different deviation types in benchmark games, where we find that stronger types typically induce better performance.
 Award ID(s):
 1761546
 Publication Date:
 NSFPAR ID:
 10290819
 Journal Name:
 Proceedings of Machine Learning Research
 ISSN:
 26403498
 Sponsoring Org:
 National Science Foundation
More Like this


The existence of simple uncoupled noregret learning dynamics that converge to correlated equilibria in normalform games is a celebrated result in the theory of multiagent systems. Specifically, it has been known for more than 20 years that when all players seek to minimize their internal regret in a repeated normalform game, the empirical frequency of play converges to a normalform correlated equilibrium. Extensiveform games generalize normalform games by modeling both sequential and simultaneous moves, as well as imperfect information. Because of the sequential nature and presence of private information in the game, correlation in extensiveform games possesses significantly different properties than its counterpart in normalform games, many of which are still open research directions. Extensiveform correlated equilibrium (EFCE) has been proposed as the natural extensiveform counterpart to the classical notion of correlated equilibrium in normalform games. Compared to the latter, the constraints that define the set of EFCEs are significantly more complex, as the correlation device must keep into account the evolution of beliefs of each player as they make observations throughout the game. Due to that significant added complexity, the existence of uncoupled learning dynamics leading to an EFCE has remained a challenging open research question for a longmore »

Regret minimization has proved to be a versatile tool for tree form sequential decision making and extensiveform games. In large twoplayer zerosum imperfectinformation games, mod ern extensions of counterfactual regret minimization (CFR) are currently the practical state of the art for computing a Nash equilibrium. Most regretminimization algorithms for treeform sequential decision making, including CFR, require (i) an exact model of the player’s decision nodes, observation nodes, and how they are linked, and (ii) full knowledge, at all times t, about the payoffs—even in parts of the decision space that are not encountered at time t. Recently, there has been growing interest towards relaxing some of those restric tions and making regret minimization applicable to settings for which reinforcement learning methods have traditionally been used—for example, those in which only blackbox access to the environment is available. We give the first, to our knowl edge, regretminimization algorithm that guarantees sublinear regret with high probability even when requirement (i)—and thus also (ii)—is dropped. We formalize an online learning setting in which the strategy space is not known to the agent and gets revealed incrementally whenever the agent encoun ters new decision points. We give an efficient algorithm that achieves O(T 3/4)more »

We present a framework that incorporates the principle of bounded rationality into dynamic stochastic pursuitevasion games. The solution of a stochastic game is generally characterized by its (Nash) equilibria in feedback form, whose calculation may require extensive computational resources. In this paper, the agents are modeled as bounded rational entities with limited computational capabilities. We illustrate the proposed framework by applying it to a pursuitevasion game between two aerial vehicles in a stochastic wind field. We show how such a game may be discretized and properly analyzed by casting it as an iterative sequence of finitestate Markov Decision Processes (MDPs). Leveraging tools and algorithms from the cognitive hierarchy theory (“levelk thinking”) we compute the solution of the ensuing discrete game, while taking into consideration the rationality level of each agent. We also present an online algorithm for each agent to infer its opponent's rationality level.

The existence of simple, uncoupled noregret dynamics that converge to correlated equilibria in normalform games is a celebrated result in the theory of multiagent systems. Specifically, it has been known for more than 20 years that when all players seek to minimize their internal regret in a repeated normalform game, the empirical frequency of play converges to a normalform correlated equilibrium. Extensiveform (that is, treeform) games generalize normalform games by modeling both sequential and simultaneous moves, as well as private information. Because of the sequential nature and presence of partial information in the game, extensiveform correlation has significantly different properties than the normalform counterpart, many of which are still open research directions. Extensiveform correlated equilibrium (EFCE) has been proposed as the natural extensiveform counterpart to normalform correlated equilibrium. However, it was currently unknown whether EFCE emerges as the result of uncoupled agent dynamics. In this paper, we give the first uncoupled noregret dynamics that converge to the set of EFCEs in nplayer generalsum extensiveform games with perfect recall. First, we introduce a notion of trigger regret in extensiveform games, which extends that of internal regret in normalform games. When each player has low trigger regret, the empirical frequency of playmore »