INTRODUCTION

Nothing takes place in the world whose meaning is not that of some maximum or minimum.

(Leonhard Euler)

Major global events shaped by large populations in social media, such as the Arab Spring, the Black Lives Matter movement, and the fake news controversy during the 2016 U.S. presidential election, provide significant impetus for devising new models that account for macroscopic population behavior resulting from the aggregate decisions and actions taken by all individuals (Howard et al., 2011;Anderson & Hitlin, 2016;Silverman, 2016). Just as physical systems behave according to the principle of least action, to which Euler's statement alludes, population behavior consists of individual actions that may be optimal with respect to some objective. The increasing usage of social media in modern societies lends plausibility to this hypothesis (Perrin, 2015), since the availability of information enables individuals to plan and act based on their observations of the global population state. For example, a population's behavior directly affects the ranking of a set of trending topics on social media, represented by the global population distribution over topics, while each user's observation of this global state influences their choice of the next topic in which to participate, thereby contributing to future population behavior (Twitter, 2017). In general, this feedback may be present in any system where the distribution of a large population over a state space is observable (or partially observable) by each individual, whose behavior policy generates actions given such observations. This motivates multiple criteria for a model of population behavior that is learnable from real data:

1. The model captures the dependency between population distribution and their actions. 2. It represents observed individual behavior as optimal for some implicit reward.

3. It enables prediction of future population distribution given measurements at previous times.

We present a mean field game (MFG) approach to address the modeling and prediction criteria. Mean field games originated as a branch of game theory that provides tractable models of large agent populations, by considering the limit of N -player games as N tends to infinity (Lasry & Lions, 2007). In this limit, an agent population is represented via their distribution over a state space, and each agent's optimal strategy is informed by a reward that is a function of the population distribution and their aggregate actions. The stochastic differential equations that characterize MFG can be specialized to many settings: optimal production rate of exhaustible resources such as oil among many producers (Guéant et al., 2011); optimizing between conformity to popular opinion and consistency with one's initial position in opinion networks (Bauso et al., 2016); and the transition between competing technologies with economy of scale (Lachapelle et al., 2010). Representing agents as a distribution means that MFG is scalable to arbitrary population sizes, enabling it to simulate real-world phenomenon such as the Mexican wave in stadiums (Guéant et al., 2011).

As the model detailed in Section 3 will show, MFG naturally addresses the modeling criteria in our problem context while overcoming limitations of alternative predictive methods. For example, time series analysis builds predictive models from data, but these models are incapable of representing any motivation (i.e. reward) that may produce a population's behavior policy. Alternatively, methods that employ the underlying population network structure have assumed that nodes are only influenced by a local neighborhood, do not account for a global state, and may face difficulty in explaining events as the result of any implicit optimization. (Farajtabar et al., 2015;De et al., 2016).

MFG is unique as a descriptive model whose solution tells us how a system naturally behaves according to its underlying optimal control policy. This observation enables us to draw a connection with the framework of Markov decision processes (MDP) and reinforcement learning (RL) (Sutton & Barto, 1998). The crucial difference from a traditional MDP viewpoint is that we frame the problem as MFG model inference via MDP policy optimization: we use the MFG model to describe natural system behavior by solving an associated MDP, without imposing any control on the system. MFG offers a computationally tractable framework for adapting inverse reinforcement learning (IRL) methods (Ng & Russell, 2000;Ziebart et al., 2008;Finn et al., 2016), with flexible neural networks as function approximators, to learn complex reward functions that may explain behavior of arbitrarily large populations. In the other direction, RL enables us to devise a data-driven method for solving an MFG model of a real-world system for temporal prediction. While research on the theory of MFG has progressed rapidly in recent years, with some examples of numerical simulation of synthetic toy problems, there is a conspicuous absence of scalable methods for empirical validation (Lachapelle et al., 2010;Achdou et al., 2012;Bauso et al., 2016). Therefore, while we show how MFG is well-suited for the specific problem of modeling population behavior, we also demonstrate a general data-driven approach to MFG inference via a synthesis of MFG and MDP.

Our main contributions are the following. We propose a data-driven approach to learn an MFG model along with its reward function, showing that research in MFG need not be confined to toy problems with artificial reward functions. Specifically, we derive a discrete time graph-state MFG from general MFG and provide detailed interpretation in a real-world setting (Section 3). Then we prove that a special case can be reduced to an MDP and show that finding an optimal policy and reward function in the MDP is equivalent to inference of the MFG model (Section 4). Using our approach, we empirically validate an MFG model of a population's activity distribution on social media, achieving significantly better predictive performance compared to baselines (Section 5). Our synthesis of MFG with MDP has potential to open new research directions for both fields.

RELATED WORK

Mean field games originated in the work of Lasry & Lions (2007), and independently as stochastic dynamic games in Huang et al. (2006), both of which proposed mean field problems in the form of differential equations for modeling problems in economics and analyzed the existence and uniqueness of solutions. Guéant et al. (2011) provided a survey of MFG models and discussed various applications in continuous time and space, such as a model of population distribution that informed the choice of application in our work. Even though the MFG framework is agnostic towards the choice of cost function (i.e. negative reward), prior work make strong assumptions on the cost in order to attain analytic solutions. We take a view that the dynamics of any game is heavily impacted by the reward function, and hence we propose methods to learn the MFG reward function from data.

Discretization of MFGs in time and space have been proposed (Gomes et al., 2010;Achdou et al., 2012;Guéant, 2015), serving as the starting point for our model of population distribution over discrete topics; while these early work analyze solution properties and lack empirical verification, we focus on algorithms for attaining solutions in real-world settings. Related to our application case, prior work by Bauso et al. (2016) analyzed the evolution of opinion dynamics in multi-population environments, but they imposed a Gaussian density assumption on the initial population distribution and restrictions on agent actions, both of which limit the generality of the model and are not assumed in our work. There is a collection of work on numerical finite-difference methods for solving continuous mean field games (Achdou et al., 2012;Lachapelle et al., 2010;Carlini & Silva, 2014). These methods involve forward-backward or Newton iterations that are sensitive to initialization and have inherent computational challenges for large real-valued state and action spaces, which limit these methods to toy problems and cannot be scaled to real-world problems. We overcome these limitations by showing how the MFG framework enables adaptation of RL algorithms that have been successful for problems involving unknown reward functions in large real-world domains.

In reinforcement learning, there are numerous value-and policy-based algorithms employing deep neural networks as function approximators for solving MDPs with large state and action spaces (Mnih et al., 2013;Silver et al., 2014;Lillicrap et al., 2015). Even though there are generalizations to multi-agent settings (Hu et al., 1998;Littman, 2001;Lowe et al., 2017), the MDP and Markov game frameworks do not easily suggest how to represent systems involving thousands of interacting agents whose actions induce an optimal trajectory through time. In our work, mean field game theory is the key to framing the modeling problem such that RL can be applied.

Methods in unknown MDP estimation and inverse reinforcement learning aim to learn an optimal policy while estimating an unknown quantity of the MDP, such as the transition law (Burnetas & Katehakis, 1997), secondary parameters (Budhiraja et al., 2012), and the reward function (Ng & Russell, 2000). The maximum entropy IRL framework has proved successful at learning reward functions from expert demonstrations (Ziebart et al., 2008;Boularias et al., 2011;Kalakrishnan et al., 2013). This probabilistic framework can be augmented with deep neural networks for learning complex reward functions from demonstration samples (Wulfmeier et al., 2015;Finn et al., 2016).

Our MFG model enables us to extend the sample-based IRL algorithm in Finn et al. (2016) to the problem of learning a reward function under which a large population's behavior is optimal, and we employ a neural network to process MFG states and actions efficiently.

MEAN FIELD GAMES

We begin with an overview of a continuous-time mean field games over graphs, and derive a general discrete-time graph-state MFG (Guéant, 2015). Then we give a detailed presentation of a discretetime MFG over a complete graph, which will be the focus for the rest of this paper.

MEAN FIELD GAMES ON GRAPHS

Let G = (V, E) be a directed graph, where the vertex set V = {1, . . . , d} represents d possible states of each agent, and E ⊆ V ×V is the edge set consisting of all possible direct transition between states (i.e., a agent can hop from i to j only if (i, j) ∈ E). For each node i ∈ V, define

be the density (proportion) of agent population in state i at time t, and π(t) := (π 1 (t), . . . , π d (t)). Population dynamics are generated by right stochastic matrices

Moreover, we have a value function V i (t) of state i at time t, and a reward function r i (π(t), P i (t))foot_0 , quantifying the instantaneous reward for agents in state i taking transitions with probability P i (t) when the current distribution is π(t). We are mainly interested in a discrete time graph state MFG, which is derived from a continuous time MFG by the following proposition. Appendix A provides a derivation from the continuous time MFG.

Proposition 1. Under a semi-implicit discretization scheme with unit time step labeled by n, the backward Hamilton-Jacobi-Bellman (HJB) equation and the forward Fokker-Planck equation for each i ∈ {1, . . . , d} and n = 0, . . . , N -1 in a discrete time graph state MFG are given by: (HJB)

)). We focus on the complete graph in this paper, as the methodology can be readily applied to general directed graphs. While Section 4 will show a connection between MFG and MDP, we note here that a "state" in the MFG sense is a node in V and not an MDP state. 2 We now interpret the model using the example of evolution of user activity distribution over topics on social media, to provide intuition and set the context for our real-world experiments in Section 5. Independent of any particular interpretation, the MFG approach is generally applicable to any problem where population size vastly outnumbers a set of discrete states.

• Population distribution π n ∈ ∆ d-1 for n = 0, . . . , N . Each π n is a discrete probability distribution over d topics, where π n i is the fraction of people who posted on topic i at time n. Although a person may participate in more than one topic within a time interval, normalization can be enforced by a small time discretization or by using a notion of "effective population size", defined as population size multiplied by the max participation count of any person during any time interval. π 0 is a given initial distribution.

• Transition matrix P n ∈ S(G). P n ij is the probability of people in topic i switching to topic j at time n, so we refer to P n i as the action of people in topic i. P n generates the forward equation

. . , d}. This is the reward received by people in topic i who choose action P n i at time n, when the distribution is π n . In contrast to previous work, we learn the reward function from data (Section 4.1). We make a locality assumption: reward for i depends only on P n i , not on the entire P n , which means that actions by people in j = i have no instantaneous effect on the reward for people in topic i.

i is the expected maximum total reward of being in topic i at time n. A terminal value V N is given, which we set to zero to avoid making any assumption on the problem structure beyond what is contained in the learned reward function.

• Average reward e i (π, P, V ), for i ∈ {1, . . . , d} and V ∈ R d and P ∈ S(G). This is the average reward received by agents at topic i when the current distribution is π, action P is chosen, and the subsequent expected maximum total reward is V . For a general r ij (π, P ), it is defined as:

Intuitively, agents want to act optimally in order to maximize their expected total average reward. For P ∈ S(G) and a vector q ∈ S i (G), define P(P, i, q) to be the matrix equal to P , except with the i-th row replaced by q. Then a Nash maximizer is defined as follows: Definition 1. A right stochastic matrix P ∈ S(G) is a Nash maximizer of e(π, P, V ) if, given a fixed π ∈ ∆ d-1 and a fixed V ∈ R d , there is e i (π, P, V ) ≥ e i (π, P(P, i, q), V ) (5) for any i ∈ {1, . . . , d} and any q ∈ S i (G).

The rows of P form a Nash equilibrium set of actions, since for any topic i, the people in topic i cannot increase their reward by unilaterally switching their action from P i to any q. Under Definition 1, the value function of each topic i at each time n satisfies the optimality criteria:

A solution of the MFG is a sequence of pairs {(π n , V n )} n=0,...,N satisfying optimality criteria (6) and forward equation (3).

INFERENCE OF MFG VIA MDP OPTIMIZATION

A Markov decision process is a well-known framework for optimization problems. We focus on the discrete time MFG in Section 3.2 and prove a reduction to a finite-horizon deterministic MDP, whose state trajectory under an optimal policy coincides with the forward evolution of the MFG. This leads to the essential insight that solving the optimization problem of an MDP is equivalent to solving an MFG that describes population behavior. This connection will enable us to apply efficient inverse RL methods, using measured population trajectories, to learn an MFG model along with its reward function in Section 4.1. The MDP is constructed as follows: Definition 2. A finite-horizon deterministic MDP for a discrete time MFG over a complete graph is defined as:

, the population distribution at time n.

• Actions: P n ∈ S(G), the transition probability matrix at time n.

• Reward: R(π n , P n ) :=

• Finite-horizon state transition, given by Eq (3): ∀n ∈ {0, . . . , N -1} :

The value function of a solution to the discrete time MFG over a complete graph defined by optimality criteria (6) and forward equation (3) is a solution to the Bellman optimality equation of the MDP in Definition 2.

Proof. Since r ij depends on P n only through row P n i , optimality criteria 6 can be written as

We now define V * (π n ) as follows and show that it is the value function of the constructed MDP in Definition 2 by verifying that it satisfies the Bellman optimality equation:

which is the Bellman optimality equation for the MDP in Definition 2.

Corollary 1. Given a start state π 0 , the state trajectory under the optimal policy of the MDP in Definition 2 is equivalent to the forward evolution part of the solution to the MFG.

Proof. Under the optimal policy, equations 11 and 8 are satisfied, which means the matrix P generated by the optimal policy at any state π n is the Nash maximizer matrix. Therefore, the state trajectory {π n } n=0,...,N is the forward part of the MFG solution.

REINFORCEMENT LEARNING SOLUTION FOR MFG

MFG provides a general framework for addressing the problem of modeling population dynamics, while the new connection between MFG and MDP enables us to apply inverse RL algorithms to solve the MDP in Definition 2 with unknown reward. In contrast to previous MFG research, most of which impose reward functions that are quadratic in actions and logarithmic in the state distribution (Guéant, 2009;Lachapelle et al., 2010;Bauso et al., 2016), we learn a reward function using demonstration trajectories measured from actual population behavior, to ground the MFG representation of population dynamics on real data.

We leverage the MFG forward dynamics (Eq 3) in a sample-based IRL method based on the maximum entropy IRL framework (Ziebart et al., 2008). From this probabilistic viewpoint, we minimize the relative entropy between a probability distribution p(τ ) over a space of trajectories T := {τ i } i and a distribution q(τ ) from which demonstrated expert trajectories are generated (Boularias et al., 2011). This is related to a path integral IRL formulation, where the likelihood of measured optimal trajectories is evaluated only using trajectories generated from their local neighborhood, rather than uniformly over the whole trajectory space (Kalakrishnan et al., 2013). Specifically, making no assumption on the true distribution of optimal demonstration other than matching of reward expectation, we posit that demonstration trajectories τ i = (π 0 , P 1 , . . . , π N -1 , P N -1 ) i are sampled from the maximum entropy distribution (Jaynes, 1957):

where R W (τ ) = n R W (π n , P n ) is the sum of reward of single state-action pairs over a trajectory τ , and W are the parameters of the reward function approximator (derivation in Appendix E).

Intuitively, this means that trajectories with higher reward are exponentially more likely to be sampled. Given M sample trajectories τ j ∈ D samp from k distributions F 1 (τ ), . . . , F k (τ ), an unbiased estimator of the partition function Z = exp(R W (τ ))dτ using multiple importance sampling is Ẑ := 1 M τj z j exp(R W (τ j )) (Owen & Zhou, 2000), where importance weights are

-1 (derivation in Appendix F). Each action matrix P is sampled from a stochastic policy F k (P ; π, θ) (overloading notation with F (τ )), where π is the current state and θ the policy parameter. The negative log likelihood of L demonstration trajectories τ i ∈ D demo is:

We build on Guided Cost Learning (GCL) in Finn et al. (2016) (Alg 1) to learn a deep neural network approximation of R W (π, P ) via stochastic gradient descent on L(W ), and learn a policy F (P ; π, θ) using a simple actor-critic algorithm (Sutton & Barto, 1998). In contrast to GCL, we employ a combination of convolutional neural nets and fully-connected layers to process both the action matrix P and state vector π efficiently in a single architecture (Appendix C), analogous to how Lillicrap et al. (2015) handle image states in Atari games. Due to our choice of policy parameterization (described below), we also set importance weights to unity for numerical stability. These implementation choices result in successful learning of a reward representation (Fig 1).

Our forward MDP solver (Alg 2) performs gradient ascent on the policy's expected start value E[v(π 0 )|F (P ; π, θ)] w.r.t. θ, to find successively better policies F k (P ; π, θ). We construct the joint distribution F (P ; π, θ) informed by domain knowledge about human population behavior on social media, but this does not reduce the generality of the MFG framework since it is straightforward to employ flexible policy and value networks in a DDPG algorithm when intuition is not available (Silver et al., 2014;Lillicrap et al., 2015). Our joint distribution is d instances of a d-dimensional Dirichlet distribution, each parameterized by an α i ∈ R d + . Each row P i is sampled from

where B(•) is the Beta function and α i j is defined using the softplus function α i j (π, θ) := ln(1 + exp{θ(π j -π i )}), which is a monotonically increasing function of the population density difference π j -π i . In practice, a constant scaling factor c ∈ R can be applied to α for variance reduction. Finally, we let F (P n ; π n , θ) = d i=1 f (P n i ; α i (π n , θ)) denote the parameterized policy, from which P n is sampled based on π n , and whose logarithmic gradient ∇ θ ln(F ) can be used in a policy gradient algorithm. We learned an approximate value function V (π; w) as a baseline for variance reduction, approximated as a linear combination of all polynomial features of π up to second order, with parameter w (Sutton et al., 2000). in reinforcement learning. Our method is scalable to arbitrarily large populations, because the MFG framework represents population density rather than individual agents, while the representations are linear in the number of MFG states and quadratic in the transition matrix. Our experiments on real data show that MFG is a powerful framework for learning a reward and policy that can predict trajectories of a real world population more accurately than alternatives. Even with a simple policy parameterization designed via some domain knowledge, our method attained superior performance on test data. It motivates exploration of flexible neural networks for more complex applications.

An interesting extension is to develop an efficient method for solving the discrete time MFG in a more general setting, where the reward at each state i is coupled to the full population transition matrix. Our work also opens the path to a variety of real-world applications, such as a synthesis of MFG with models of social networks at the level of individual connections to construct a more complete model of social dynamics, and mean field models of interdependent systems that may display complex interactions via coupling through global states and reward functions.

A PROOF OF PROPOSTION 1

Given the definitions in Section 3.1, a mean field game is defined by a Hamilton-Jacobi-Bellman (HJB) equation evolving backwards in time and a Fokker-Planck equation evolving forward in time. The continuous-time Hamilton-Jacobi-Bellman (HJB) equation on G is

where r i (π, P i ) is the reward function, and V i (t) is the value function of state i at time t. Note that the reward function r i (π(t), P i (t)) is often presented as -c i (π(t), P i (t)) for some cost function c i (π(t), P i (t)) in the MFG context, and similarly for V i (t). In addition, we set r i (π(t),

e. P (t) must be a valid transition matrix). For any fixed π(t), let H i (π(t), •) be the Legendre transform of c i (π(t), •) defined by

Then the HJB equation ( 15) is an analogue to the backward equation in mean field games

where

is the dual variable of P i . We can discretize (15) using a semi-implicit scheme with unit time step labeled by n to obtain

Rearranging ( 18) yields the discrete time HJB equation over a graph (19)

The forward evolving Fokker-Planck equation for the continuous-time graph-state MFG is given by

where

where ∂ ui H j (π, u) is the partial derivative w.r.t. the coordinate corresponding to the i-th index of the argument u ∈ R | V- j | . We can set Q ji (t) = 0 for all (j, i) / ∈ E, so that Q(t) := [Q ji (t)] can be regarded as the d-by-d infinitesimal generator matrix of states π(t), and hence (20) can be written as π ′ (t) = π(t)Q(t), where π(t) ∈ R d is a row vector. Then an Euler discretization of (20) with unit time step reduces to π n+1 -π n = π n Q n , which can be written as

where

If the graph G is complete, meaning E = {(i, j) : 1 ≤ i, j ≤ d}, then the summation is taken over j = 1, . . . , d. For ease of presentation, we only consider the complete graph in this paper, as all derivations can be carried out similarly for general directed graphs. A solution of a mean field game defined by ( 19) and ( 22) is a collection of V n i and π n i for i = 1, . . . , d and n = 0, . . . , N .

B ALGORITHMS

We learn a reward function and policy using an adaptation of GCL (Finn et al., 2016) in Alg 1 and a simple actor-critic Alg 2 (Sutton & Barto, 1998) as a forward RL solver.

Algorithm 1 Guided cost learning 1: procedure GUIDED COST LEARNING 2:

Initialize F 0 (P ; π, θ) as random policy and reward network weights W 0 while Avg πi,Pi∼Ddemo (R W t (π i , P i ) -R W t-1 (π i , P i )) > dR do for episodes s = 1, . . . , S do 4:

Sample initial distribution π 0 from {π 0 } M days 5:

for time step n = 0, . . . , N -1 do 6:

Sample action P n ∼ F (P ; π n , θ)

7:

Generate π n+1 using Eq 3 8:

Receive reward R W (π n , P n ) 9:

δ ← R + V (π n+1 ; w) -V (π n ; w) 10:

w ← w + ξδ∇ w V (π n ; w)

11:

θ ← θ + βδ∇ θ log(F (P ; π n , θ))

12:

end for 13:

end for 14: end procedure C REWARD NETWORK Our reward network uses two convolutional layers to process the 15 × 15 action matrix P , which is then flattened and concatenated with the state vector π and processed by two fully-connected layers regularized with L1 and L2 penalties and dropout (probability 0.6). The first convolutional layer zero-pads the input into a 19 × 19 matrix and convolves one filter of kernel size 5 × 5 with stride 1 and applies a rectifier nonlinearity. The second convolutional layer zero-pads its input into a 17 × 17 matrix and convolves 2 filters of kernel size 3 × 3 with stride 1 and applies a rectifier nonlinearity. The fully connected layers have 8 and 4 hidden rectifier units respectively, and the output is a single fully connected tanh unit. All layers were initialized using the Xavier normal initializer in Tensorflow.

D EXPERIMENT DETAILS

By default, Twitter users in a certain geographical region primarily see the trending topics specific to that region (Twitter, 2017). This experiment focused on the population and trending topics in the city of Atlanta in the U.S. state of Georgia. First, a set of 406 active users were collected to form the fixed population. This was done by collecting a set of high-visibility accounts in Atlanta (e.g. the Atlanta Falcons team), gathering all Twitter users who follow these accounts, filtering for those whose location was set to Atlanta, and filtering for those who responded to least two trending topics within four days.

Data collection proceeded as follows for 27 days: at 9am of each day, a list of the top 14 trending topics on Twitter in Atlanta was recorded; for each hour until midnight, for each topic, the number of users who responded to the topic and the transition counts among topics within the past hour was recorded. Whether or not a user responded to a topic was determined by checking for posts by the user containing unique words for that topic; the "hashtag" convention of trending topics on Twitter reduces the likelihood of false positives. The hourly count of people who did not respond to any topic was recorded as the count for a "null topic". Although some users may respond to more than one topic within each hour, the data shows that this is negligible, and a shorter time interval can be used to reduce this effect. The result of data collection is a set of trajectories, one trajectory per day, where each trajectory consists of hourly measurements of the population distribution over d = 15 topics and their transition matrix over N = 16 hours.

The training set consists of trajectories {π 0,m , P 0,m , . . . , P N -2,m , π N -1,m } m=1,...,M over the first M = 21 days. MFG uses the initial distribution π 0 of each day, along with the transition equation of the constructed MDP and the policy F (P ; π n , θ), to produce complete trajectories for training (Alg 2 lines 4,6,7). In contrast, VAR and RNN are supervised learning methods and they use all measured distributions. RNN employs a simple recurrent unit with ReLU as nonlinear activation and weight matrix of dimension d × d. VAR was implemented using the Statsmodels module in Python, with order 18 selected via random sub-sampling validation with validation set size 5 (Seabold & Perktold, 2010). For prediction accuracy, all three methods were evaluated against data from 6 held-out test days. Table 1 shows parameters of Alg 2 and 1.

E MAXIMUM ENTROPY DISTRIBUTION

Given a finite set of trajectories {τ i } i , where each trajectory is a sequence of state-action pairs τ i = (s i1 , a i1 , . . . , ). Suppose each trajectory τ i has an unknown probability p i . The entropy of the probability distribution is H =i p i ln(p i ). In the continuous case, we write the differential entropy: The first constraint says: the expected reward over all trajectories is equal to an empirical measurement µ r . We write the Lagrangian L: where Z := e -λ2r(τ ) dτ . Then the constant λ 2 is determined by: µ r = p(τ )r(τ )dτ = 1 Z(λ 2 ) e -λ2r(τ ) r(τ )dτ = -∂ ∂λ 2 ln(Z(λ 2 ))

F MULTIPLE IMPORTANCE SAMPLING

We show how multiple importance sampling (Owen & Zhou, 2000) can be used to estimate the partition function in the maximum entropy IRL framework. The problem is to estimate Z := f (x)dx.

Let p 1 , . . . , p m be m proposal distributions, with n j samples from the j-th proposal distribution, so that samples can be denoted X ij for i = 1, . . . , n j and j = 1, . . . , m. Let w j (x) for j = 1, . . . , m satisfy 0 ≤ w j (x) ≤ m j=1 w j (x) = 1

Then define the estimator

Let S(p j ) = {x | p j (x) > 0} be the support of p j and S(w j ) = {x | w j (x) > 0} be the support of w j , and let them satisfy S(w j ) ⊂ S(p j ). Under these assumptions:

In particular, choose where n = m j=1 n j is the total count of samples. Further assuming that samples are drawn uniformly from all proposal distributions, so that n j = n k = n/m for all j, k ∈ {1, . . . , m}, the expression for Ẑ reduces to the form used in Eq 13: