<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>Statistical Challenges in Agent-Based Modeling</title></titleStmt>
			<publicationStmt>
				<publisher></publisher>
				<date>07/03/2021</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10310772</idno>
					<idno type="doi">10.1080/00031305.2021.1900914</idno>
					<title level='j'>The American Statistician</title>
<idno>0003-1305</idno>
<biblScope unit="volume">75</biblScope>
<biblScope unit="issue">3</biblScope>					

					<author>David L. Banks</author><author>Mevin B. Hooten</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[Agent-based models (ABMs) are popular in many research communities, but few statisticians have contributed to their theoretical development. They are models like any other models we study, but in general, we are still learning how to fit ABMs to data and how to make quantified statements of uncertainty about the outputs of an ABM. ABM validation is also an underdeveloped area that is ripe for new statistical developments. In what follows, we lay out the research space and encourage statisticians to address the many research issues in the ABM ambit.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1.">Introduction</head><p>Agent-based models (ABMs) are simulators used to study interactions among artificial entities (agents) that are governed by rules. The goal with ABMs is o en to understand emergent behavior based on a set of explicit rules describing how elements of a system interact. O en complex phenomena arise from simple rules.</p><p>ABMs are used in many different disciplines. Examples include:</p><p>&#8226; Weather forecasting, in which the agents could be viewed as cubic kilometers of atmosphere that exchange pressure, temperature and humidity with their neighbors according to standard physical laws <ref type="bibr">(Simmonds, G&#243;mez, and Ledezma 2019)</ref>. One can initialize the agents with specific values, and then watch a weather front roll across a continent. &#8226; Transportation, in which agents are vehicles that traverse a road way, obeying rules that determine spacing, speed, destination, and route choice. TRANSIMS is a famous early ABM. The goal is to identify points of high congestion, or to perform "what if?" experiments that study the impact of, for example, a lane closure. &#8226; Sociology, in which agents are "people" with various characteristics (e.g., age, gender, and education) who form friendship networks according to theories of homophily or heterophily <ref type="bibr">(Snijders 1996)</ref>. The goal might be to estimate the amount of homophily that leads to networks whose connectivity patterns resemble those of human networks. &#8226; Epidemiology, in which infected agents interact with each other to spread a virtual disease. The goal could be to determine what kinds of public health response (handwashing campaigns, social distancing, school closure, and quarantine) are effective for diseases with specific infectiousness and incubation period <ref type="bibr">(Perez and Dragicevic 2009;</ref><ref type="bibr">Frias-Martinez, Williamson, and Frias-Martinez 2011)</ref>.</p><p>CONTACT David L. Banks banks@stat.duke.edu Department of Statistical Science, Duke University.</p><p>ABMs have become a standard tool in many fields, but their use is still nascent in statistics <ref type="bibr">(Hooten, Wikle, and Schwob 2020)</ref>. They are models like many others that we use, but statisticians have not yet agreed on a theory and methodology for estimating their parameters from data or for making quantified statements of uncertainty about ABM predictions.</p><p>There are, of course, a few exceptions. <ref type="bibr">Hooten and Wikle (2010)</ref> developed an ABM to infer the probability of rabies in Connecticut townships, <ref type="bibr">and Hunter, Mac Namee, and Kelleher (2018)</ref> use an ABM to describe H1N1 spread. But these are isolated studies that make special assumptions and that address relatively simple situations.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1.1.">History of ABMs</head><p>The ABM perspective dates back to the 1940s, with work on cellular automata by John von Neumann and Stanislaw Ulam <ref type="bibr">(Von Neumann et al. 1966)</ref>. The most famous of these automata was J. H. Conway's Game of Life <ref type="bibr">(Gardner 1970)</ref>. The agents are squares on a sheet of graph paper in the Game of Life. The squares may be colored either white or black and may change color from one time step to another. A black square with fewer than two black neighbors (out of eight) turns white in the next time step; a black square with two or three black neighbors stays black; a black square with more than three black neighbors turns white; and any white cell with three black neighbors turns black. This rule set generated complex and sometimes persistent patterns, which has made the game a longstanding object of mathematical study <ref type="bibr">(Berlekamp, Conway, and Guy 2004)</ref>.</p><p>The next step in the evolution of ABMs was the development of theory for interactive particle systems. Prominent probabilists involved in this work include Frank Spitzer and David Griffeath, who used methods from statistical mechanics to study system dynamics and phenomena such as phase changes <ref type="bibr">(Griffeath 1993</ref>).</p><p>In the 1990s, ABMs broke away from mathematics and probability, becoming widely used in biology, ecology, economics, epidemiology, military strategy, political science, sociology, and transportation studies. This popularity was the result of many factors. One factor was that computation became both faster and more accepted as a scientific tool of inquiry. The increased computational power made it possible to simulate realistically complex scenarios. Another factor was that the reasoning process that underlies the creation of an ABM enabled domain experts to build models in a new way, without having to master sophisticated mathematics. A third factor was that several highprofile applications grabbed the imagination of those research communities.</p><p>Looking ahead, it is clear that ABMs will remain a popular tool across a wide range of scientific fields. They are also becoming accepted as a public policy tool, in large part because of their ability to perform "what if?" experiments. For example, the TRANSIMS ABM enables city planners to model the impact of a road closure, and possibly foresee unintended consequences. Such tools are being discussed at the national level in Sweden <ref type="bibr">(Eliasson and Taymaz 1992</ref>) and in regional urban planning <ref type="bibr">(Gilbert 2019;</ref><ref type="bibr">Li and Liu 2008, chap. 7</ref>) <ref type="bibr">Chen and Zhan (2014)</ref> used them to evaluate urban evacuation plans, and <ref type="bibr">Klabunde and Willekens (2016)</ref> used them to model international migrants.</p><p>Modern and complex ABMs typically have all or most of the following features:</p><p>&#8226; Manyagents,o enwithdifferentiatedrolesandthusdifferent rule sets. &#8226; Rules, which may be complex. Sometimes the rules are heuristic, sometimes an ABM is based on randomized rules. &#8226; Learning and adaptation. Agents learn about their environment (including other agents). Axelrod's Prisoner's Dilemma competition is an example <ref type="bibr">(Axelrod 1980</ref>). &#8226; An interaction space. This defines which agents affect each other-usually this is a model of propinquity, but for auctions it is a star-graph. &#8226; A nonagent environment. This may include initial conditions, and/or background processes.</p><p>To illustrate these ideas, the following section describes three examples of ABMs. Following that is a discussion of the inferential strategies that the statistical community might apply to ABMs, and we close with a few observations.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.">Examples</head><p>We review three case studies in what follows. These case studies illustrate ideas and issues that arise in ABM applications.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.1.">Evacuating a Building</head><p>For safety purposes, civil engineers must study how long it takes a building with a specific design to be evacuated in an emergency. ABMs are a standard tool <ref type="bibr">(Pelechano and Malkawi 2008;</ref><ref type="bibr">Zheng, Zhong, and Liu 2009)</ref>. A procedure may involve 100 ABM simulations of the evacuation and result in a histogram of the time needed for the last agent to exit the building.</p><p>In this example, the interaction space is the virtually rendered architecture of the building being studied. The agents are "people" who are randomly distributed within the building. As an example of differentiated agent roles, some of the people may be "fire marshals" who direct others to safe stairwells. The rules for most agents can be quite simple: (i) When the alarm goes off, exit by the closest stairwell. (ii) If the stairs are blocked, try the next nearest stairwell. (iii) If a fire marshal directs you to a specific stairwell, use it. (iv) Move at a prespecified rate (perhaps different for different agents). Fire marshal rules could be specified similarly and comparably simple.</p><p>The nonagent environment in this example are such things as conditions that prevent a faster agent from walking through a slower agent on the stairs, and how to handle bottlenecks at doorways. Learning occurs when an agent discovers that a stairwell is blocked, or receives instruction from a fire marshal.</p><p>In this kind of simulation, one can model actions second-bysecond and include a great deal of detail, such as the location of office furniture and agents with mobility challenges. However, as a general rule, one should avoid overcomplicating ABMs with unnecessarily granular detail.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.2.">Growing Artificial Societies</head><p>Sociologists became excited about ABMs in part because of an influential book, Growing Artificial Societies: Social Science from the Bottom Up <ref type="bibr">(Epstein and Axtell 1996)</ref>. The environment is a "sugarscape, " which is a Cartesian plane on which a resource, "sugar, " grows at a fixed rate at the lattice points. The agents are virtual people who stand at lattice points and consume the sugar until it is gone; then they search for a new lattice point at which to harvest sugar.</p><p>As an example, here are the first three of 17 rules used in <ref type="bibr">Epstein and Axtell (1996)</ref>.</p><p>1. Sugarscape Growback: At each lattice position, sugar grows back at a rate of &#945; per time interval up to the capacity of that position. 2. Agent Movement: Look out as far as vision permits in each of the four lattice directions, north, south, east, and west:</p><p>&#8226; Considering only unoccupied lattice positions, find the nearest position producing maximum welfare; &#8226; Move to the new position; &#8226; Collect all the resources at that location.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.">Agent Mating:</head><p>&#8226; Select a neighboring agent at random; &#8226; If the neighboring agent is of the opposite sex and if both agents are fertile and at least one of the agents has an empty neighboring site, then a newborn is produced by crossing over the parents' genetic and cultural characteristics; &#8226; Repeat for all neighbors.</p><p>Note that the first rule includes a tunable parameter; there are many such in the full rule list, and these are common in ABMs.</p><p>The first two rules govern search and movement; they generate simulated migratory patterns that mimic those seen in hunter-gatherer populations. The third rule for mating and reproduction produce age pyramids and population dynamics similar to those seen in humans. <ref type="bibr">Epstein and Axtell (1996)</ref> introduce a second resource, "spice, " and it, with additional rules, creates realistic barter economies. Ultimately, a total of 17 rules enable war, epidemics, division of labor, religion, and other rich behaviors. This kicked off a groundswell of interest in the sociological community, and now these ABM tools are used for many purposes, notably the study of social networks <ref type="bibr">(Snijders 1996)</ref> and illegal drug markets <ref type="bibr">(Dray et al. 2008;</ref><ref type="bibr">Romano, Lomax, and Richmond 2009)</ref>.</p><p>In this application, the agents are differentiated (male/female, buyers/sellers), the rules are surprisingly simple, agents adapt to their environment in various ways, such as by movement and consumption. The interaction space is determined by the distance to nearby agents, and the nonagent environment is the sugarscape. This ABM has a compelling balance between simple rules and complex behavior.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.3.">Disease Spread and Epidemiology</head><p>A common application of ABMs is to study disease spread <ref type="bibr">(Perez and Dragicevic 2009;</ref><ref type="bibr">Frias-Martinez, Williamson, and Frias-Martinez 2011)</ref>. These are generally complementary to differential equation models and other alternatives <ref type="bibr">(Bobashev et al. 2007;</ref><ref type="bibr">Hefley et al. 2017)</ref>, and o en enable greater realism and accuracy <ref type="bibr">(Ajelli et al. 2010)</ref>.</p><p>The general strategy is straightforward. One uses or models data on transportation flows within the geographic unit of interest (e.g., nursing home, city, nation, or world). The analyst picks disease parameters (e.g., age-specific infectiousness, incubation period, and mortality rate), and then seeds the community with an infected person, who travels within the geography, interacting with people in realistic ways. O en the interest is in the peak number of ill people, the duration of the outbreak, and the effect of mitigating measures, such as closing schools or social distancing.</p><p>In this example, the agents generally take different roles (e.g., children, physicians, people with more or less exposure). The rules mostly govern movement. The interactions are opportunities to spread the infection, behavioral adaptation includes such things as social distancing or going to a hospital, and the environment is the geography. Some ABM simulations are quite detailed, but generally the important questions can be addressed with relatively simple models.</p><p>The disease application indicates one of the limitations of ABMs. There were several ABMs for the 2014 ebola outbreak in Liberia <ref type="bibr">(Merler et al. 2015;</ref><ref type="bibr">Siettos et al. 2015;</ref><ref type="bibr">Venkatramanan et al. 2018)</ref>. Generally, such models did well in predicting how many villages would experience an outbreak in the next week, but they were poor at predicting which villages would suffer. This is reasonable, because a common transmission pattern in Liberia was someone working in Monrovia would start to feel ill, and then travel to their village so that family could care for them. The ABM could predict how many new cases would occur in the city, but it was not able to forecast the home village of the infected person.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.">Inferential Strategies</head><p>Stochastic ABMs are models just like other statistical models but, with some exceptions, we cannot write down their likelihood functions. Therefore, we o en have little theoretical guidance in estimating the parameters needed to fit the model to data, or in making quantified statements of uncertainty about model forecasts.</p><p>Sometimes ABM likelihoods are accessible, as we describe in what follows. But when the likelihood function is intractable, statisticians generally must use one of two tools. The first is statistical emulation, and the second is Approximate Bayesian Computation (ABC).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.1.">Tractable ABMs: Spread of Epidemiological and Ecological Processes</head><p>Many statistical models used in wildlife ecology to study population dynamics are individual-based and tractable using conventional statistical methods (i.e., maximum likelihood or Bayesian methods). For example, capture-recapture models are mature and provide inference about population abundance, survival, recruitment, and movement (e.g., <ref type="bibr">King et al. 2009;</ref><ref type="bibr">Hooten, Wikle, and Schwob 2020)</ref>. These models use data sources that arise from tracking (or relocating) animals over time to understand various demographic rates that can aid in the management and conservation of species. Most capture-recapture models are simple enough that they can be fit to data using conventional statistical methods. This is because they o en lack components that allow for interaction among individuals, and hence individuals are assumed to be conditionally independent, which facilitates a hierarchical model structure that is amenable to implementation.</p><p>Like the capture-recapture class of models, the rapidly growing field of animal movement modeling treats dynamic individual-level processes explicitly <ref type="bibr">(Hooten et al. 2017)</ref>. Individual-based animal movement models could be referred to as ABMs, yet rarely are (but see <ref type="bibr">Hooten et al. 2010b</ref>).</p><p>In fact, population level inference using animal movement models o en proceeds by treating individual-level parameters as random effects <ref type="bibr">(Hooten et al. 2016)</ref>. To formally scale individual-level processes up to the population level, many approaches use PDEs <ref type="bibr">(Wikle 2003;</ref><ref type="bibr">Hooten and Wikle 2008;</ref><ref type="bibr">Lu et al. 2019)</ref>. While these approaches to modeling collections of individuals have mechanistic mathematical underpinnings and have been simplified to be computationally tractable, they are o en specified in terms of an Eulerian approximation to an underlying Lagrangian ABM for individual movement <ref type="bibr">(Turchin 1998)</ref>.</p><p>Despite their sophistication, most population-level models based on individual-level animal movement processes do not account for interactions among individuals beyond density dependence. For example, <ref type="bibr">Lu et al. (2019)</ref> extended a spatiotemporal population model developed by <ref type="bibr">Williams et al. (2017)</ref> to describe the colonization of Glacier Bay by sea otters that incorporated regulated population growth based on density dependence. More complex interactions among individuals are the basis for realistic ABMs and have been accommodated in models for animal movement (e.g., <ref type="bibr">Scharf et al. 2016</ref><ref type="bibr">Scharf et al. , 2018))</ref>. These models, o en referred to as "collective" movement models (e.g., <ref type="bibr">Couzin et al. 2005;</ref><ref type="bibr">Strandburg-Peshkin et al. 2015;</ref><ref type="bibr">Russell et al. 2017)</ref> have been used for a variety of applications but are not usually treated in a formal statistical framework. Moreover, connecting other important ecological and epidemiological processes with movement models may be challenging in conventional statistical frameworks.</p><p>Motivated by <ref type="bibr">Smith et al. (2002)</ref>, who employed a cellular automata model for the spread of rabies in raccoon populations throughout Connecticut, USA, Hooten and Wikle (2010) developed a statistical ABM for the spread of ecological and epidemiological processes. Instead of using a PDE as the basis for characterizing movement, their approach relied on fundamental dispersal processes of raccoons on a regular grid with resolution matching the approximate size of Connecticut townships. However, rather than use deterministic cellular automata, they specified the entire model in a hierarchical Bayesian framework that linked movement probabilities to a gradient surface based on spatial covariates to understand barriers to and corridors for disease spread. They fit the model using conventional Bayesian computational methods such as MCMC. In fact, <ref type="bibr">Broms et al. (2016)</ref> generalized the statistical ABM to account for imperfect detection and used it to model the spread of an invasive bird in South Africa, fitting the model with JAGS <ref type="bibr">(Plummer et al. 2003)</ref>.</p><p>At the heart of these statistical ABMs is a mixture Bernoulli process for binary data <ref type="bibr">(Wikle and Hooten 2015)</ref>. For example, following <ref type="bibr">Broms et al. (2016)</ref>, to model the observed presence or absence of a species y ij (t) in region i during period j at time t, they accounted for the probability p ij (t) of detecting a species when it is present, by letting</p><p>where z i (t) represents the true latent occupancy process of the species. This type of measurement process is commonly used in the so-called occupancy models (e.g., <ref type="bibr">Hoeting, Leecaster, and Bowden 2000;</ref><ref type="bibr">MacKenzie et al. 2002;</ref><ref type="bibr">Johnson et al. 2013)</ref>.</p><p>A conditional binary model is o en specified for the latent processes z i (t) so that the probability of occupancy depends on a set of spatially referenced predictor variables as in typical species distribution models <ref type="bibr">(Hefley and Hooten 2016)</ref>. <ref type="bibr">Hooten and Wikle (2010)</ref> and <ref type="bibr">Broms et al. (2016)</ref> treated the areal units as "agents" and modeled the process using a dynamic conditional specification such as</p><p>Bern(&#968; i (t)) if z i (t -1) = 0 and z i (t -1) = 0 for all i a neighbor of i (2) where the probability &#966; i (t) represents persistence or survival of the species or disease, &#952; i (t) represents neighborhood-based dispersal probability, and &#968; i (t) represents long-distance dispersal probability. <ref type="bibr">Hooten and Wikle (2010)</ref> specified the persistence and long-distance dispersal probabilities to be homogeneous, but allowed for heterogeneous short-distance dispersal processes by allowing &#952; i,t to be the cumulative probability of immigrating individuals (or disease). This first requires an understanding of movement from the individual perspective and then aggregates that process to assess incoming individuals to new regions. Using similar principles of disease spread, <ref type="bibr">Hooten, Anderson, and Waller (2010a)</ref> modeled influenza dynamics in the continental U.S. based on multiple mechanisms of human movement.</p><p>The ABM examples described in this section were all developed to facilitate fitting statistical models to data. However, such accommodations limit their potential realism to describe more complicated processes. Thus, we o en seek inference in cases where it is computationally possible to simulate the process, but not trivial to express a joint or even conditional probability model for the data directly. This puts the statistician in an awkward position because they have access to a generative model for the data (as one does in a maximum likelihood or Bayesian setting), but they cannot evaluate a PDF or PMF associated with the model. In these cases, we must rely on some form of approximation that makes statistical inference possible. Either we must approximate the process itself which includes mechanisms in the ABM (i.e., the emulator approach), or we approximate the a Bayesian posterior distribution using an MCMC algorithm that substitutes a suitable distance metric for the likelihood (i.e., the ABC approach), or both.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2.">Emulators</head><p>Emulators are approximations (or surrogates) to complex simulation models, either deterministic or stochastic <ref type="bibr">(Gramacy 2020)</ref>. Generally, emulators use Gaussian processes or treed Gaussian processes as the model <ref type="bibr">(Gramacy and Lee 2008)</ref>. These are Bayesian procedures first proposed by <ref type="bibr">Kennedy and O'Hagan (2001)</ref>, but have been subsequently extended and developed by many others <ref type="bibr">(Bayarri et al. 2007;</ref><ref type="bibr">Higdon et al. 2008;</ref><ref type="bibr">Hooten, Anderson, and Waller 2010a;</ref><ref type="bibr">Smith 2013)</ref>.</p><p>In our case, the ABM is the complex simulator that we seek to emulate. <ref type="bibr">Farah (2014)</ref> examined such emulators in the context of an ABM for the spread of H1N1 influenza. <ref type="bibr">Heard (2014)</ref> and <ref type="bibr">Heard, Bobashev, and Morris (2014)</ref> compared emulators and ABC in the context of ABMs used to describe HIV spread and drug markets. They found that emulators were generally easier to implement and more robust.</p><p>Suppose the ABM takes input x i and produces (potentially random) outputs Y i . The ABM contains parameters (e.g., the growth rate in the sugarscape and the rules that determine agent movement) that we denote by &#952;. Then the simulation output is modeled statistically using the emulator &#951;(x i , &#952; ) as follows:</p><p>where the random term &#948;(x i ) accounts for the discrepancy between the emulator and the simulator and (x i ) is noise.</p><p>One goal is tune the parameter &#952; to find &#952; * that best calibrates the emulator to the simulator. This purpose is to calibrate the model using data assimilation, a statistical concept. A second goal is to estimate the discrepancy function; this indicates where the emulator performs poorly and may need more elaboration. This aspect enables one to infer credible regions and prediction intervals. Additional goals might include analogues of variable selection, estimation of variable importance, and insight into model dynamics.</p><p>The Kennedy-O'Hagan approach scales all inputs to the unit hypercube. Then the Bayesian version uses a Gaussian process to model the unknown function &#951;(&#8226;, &#8226;). In much of the related work, the Gaussian process is specified to have a constant mean function and a product covariance with power exponential form <ref type="bibr">(Higdon et al. 2008)</ref> cov[(x, &#952; ), (x , &#952; )] = &#955; -1 &#951; R((x, &#952;), (x ; &#961; &#951; )) where &#955; &#951; controls the marginal precision of &#951;(&#8226;, &#8226;) and &#961; &#951; controls the strength of dependency in each component of x and &#952; . It is o en useful to add a little white noise to the covariance model to account for small numerical fluctuations (from, say, adaptive meshing or convergence tolerances). The formulation of the prior is completed by specifying independent priors for the parameters controlling &#951;(&#8226;, &#8226;): the &#181;, &#955; &#951; , and &#961; &#951; .</p><p>A similar Gaussian process model is used for the discrepancy term &#948;(x). This has mean zero and covariance function cov(x, x ) = &#955; &#948; R((x, x ); &#961; &#948; ).</p><p>As before, there are technical details regarding the specification of the covariance function.</p><p>This emulator structure enables one to use Markov chain Monte Carlo (MCMC) sampling to learn the posterior distributions for critical ABM output. Specifically, one obtains posterior distributions for &#8226; the &#951;(x, &#952;), which is the hard-to-know implicit function calculated by the ABM; &#8226; the optimal calibration parameter &#952; * ; &#8226; the calibrated emulator &#951;(x, &#952; * ); &#8226; the physical system Y(x); and &#8226; the discrepancy function &#948;(x).</p><p>The last, of course, is the most interesting from the standpoint of model validation. When and where this discrepancy function is large points out missing mechanisms or poor approximations.</p><p>Iterated application of this method permits successive estimation of the discrepancy function. One winds up with an approximation to the ABM in terms of simple functions, and the accuracy of the approximation can o en be improved to the degree required by making the Gaussian process model more elaborate (e.g., by using a treed Gaussian process; <ref type="bibr">Gramacy and Lee 2008)</ref>.</p><p>In some ABM applications, one can go further in calibrating the ABM to reality. For example, in the case of weather forecasting, one has an ABM that predicts wind, humidity, and temperature in the future. Because we get to observe the weather on the following day, we can use essentially the same strategy outlined above to tune the emulator to the simulator to let us exploit the simulator to characterize real world data.</p><p>The Kennedy-O'Hagan approach is specified and implemented based on the second-moment dependence, but other emulator approaches have been developed based on the firstorder structure <ref type="bibr">(Hooten, Anderson, and Waller 2010a)</ref>. Those approaches using heterogeneous mean functions to mimic ABM processes may be faster and simpler to implement in highdimensional settings.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.3.">Approximate Bayesian Computation</head><p>ABC is an alternative to emulators when the likelihood function is unavailable. First proposed in <ref type="bibr">Rubin (1984)</ref>, it did not achieve modern form until <ref type="bibr">Tavar&#233; et al. (1997)</ref>. Although ABC antedates emulators, in many ways the methodology seems less mature.</p><p>The core idea in ABC for ABMs is that one has a sample X of data from the process that the ABM is modeling; e.g., meteorological data, in the context of a ABM that forecasts weather. The ABM has parameters &#952; and the analyst has a prior &#960;(&#952;) on those.</p><p>One then draws a value of &#952; from the prior, and conditions the ABM on that draw. Next, one generates a sample X(&#952;) from that data. For some appropriate metric d(&#8226;, &#8226;) and tolerance &#947; , the analyst checks whether d(X, X(&#952; )) &lt; &#947; . If it is within the tolerance, then &#952; is accepted accepted; if not, it is rejected. Then a new &#952; is drawn and the process repeats. The histogram of accepted values is approximately the posterior distribution that is wanted.</p><p>There are many reasons why this approach is problematic for ABMs. O en one does not have much (or any) data X, as was the case with the Sugarscape example <ref type="bibr">(Epstein and Axtell 1996)</ref>. When there are data, they can be highly multivatiate (e.g., temperature and rainfall measurements for all weather stations in a state). In that case, one must contend with the Curse of Dimensionality, or find some appropriate low-dimensional summary of the data, which in turn reduces the accuracy of the approximation. It is not obvious how to choose the metric or how to select the tolerance <ref type="bibr">(Robert et al. 2011)</ref>. For complex ABMs, it is difficult to develop a meaningful prior. Also, ABC is susceptible to pathologies <ref type="bibr">(Berger et al. 2010)</ref>.</p><p>ABC researchers have strategies for addressing all of these concerns, and ABC may be the right tool for some ABM applications. But, ABC could also more difficult to implement and less reliable that emulator methodology.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.">Model Validation</head><p>A key concern with ABMs is validating the simulation. Is the ABM elaborate enough to capture the full range of behaviors that are needed?</p><p>There are five main approaches to validation, not all of which are possible in some applications. We refer to the first as physicsbased validation, in which the ABM encodes well-understood principles that govern emerging behavior. An example would be an ABM simulation of an n-body problem, in which objects in space interact according to gravitational attraction <ref type="bibr">(Trenti and Hut 2008)</ref>. If the physics are correct and complete, then the ABM is valid.</p><p>The second approach might be termed "intelligent design. " Thoughtful experts encode all the factors that they believe are relevant, and hope they have not overlooked any important interactions. This is the most common method of validation; it is used by <ref type="bibr">Hoffer, Bobashev, and Morris (2009)</ref> to simulate a drug market and by <ref type="bibr">Farah (2014)</ref> to describe disease spread. It may be about as reliable as so ware coding in some cases, because the thought processes are similar.</p><p>The third approach is face validity, and it is a true validation protocol. The designer runs the ABM with preselected inputs, ideally inputs for which there are known real-world outcomes, and evaluates the plausibility of the outputs. Such ABM testing is commonly used for battlefield simulation by the Defense Modeling and Simulation Office and the Naval Postgraduate School, among many others <ref type="bibr">(Cioppa, Lucas, and Sanchez 2004)</ref>.</p><p>The fourth method is more rare. The analyst compares the results from the ABM to predictions from another model (which may be an independently coded ABM, or some other forecast). One example is to compare an ABM model for disease spread to, say, the Kermack-McKendrick model, which employs a set of coupled ordinary differential equations to describe an epidemic curve <ref type="bibr">(Brauer 2005)</ref>. If these agree, the ABM seems sound.</p><p>Finally, in some cases, it is possible to compare the ABM outputs to real-world observations, as in weather forecasting <ref type="bibr">(Simmonds, G&#243;mez, and Ledezma 2019)</ref>. This is the strongest form of validation, but it may still be unreliable if the analyst intends to use the ABM for inputs not previously seen (e.g., predicting weather outcomes from climate change).</p><p>As an example of the difficulty of model validation, consider the high-school social network of romantic relationships studied by <ref type="bibr">Bearman, Moody, and Stovel (2004)</ref> shown in Figure <ref type="figure">1</ref>. These sociologists found that racial and smoking homophily predicted relationships, as did gender heterophily.</p><p>However, when they coded an ABM to simulate similar networks, conditioning on the number of edges and the estimated parameters in their logit model, the outputs looked like tangled balls of yarn rather than the filamentary structure seen in Figure <ref type="figure">1</ref>. Then they realized that their data contain very few 4cycles (a swap, in which two couples exchange partners). When they reran their ABM with an additional condition to exclude 4cycles, the filamentary structure emerged. This, of course, does not prove that their ABM is correct, but illustrates the challenges of validation.</p><p>As a final comment on model validation, it is important to consider the intrinsic dimension of the ABM. Some analysts create elaborate ABMs, cluttered with code that is irrelevant to the emergent behavior. But that is not as great a problem as an ABM that omits structure needed to simulate the phenomenon, as with the initial ABM in <ref type="bibr">Bearman, Moody, and Stovel (2004)</ref>.</p><p>Every ABM has a set of parameters. The intrinsic parameters are those that control the outputs, and the rest affect the output to a lesser degree. For example, in an ABM for disease spread, it may be that the ABM models transmission in homes, at work, at the grocery store, at the laundry, and so forth. But it is plausible that the parameters driving transmission at the laundry are largely irrelevant to predicting the course of the epidemic.</p><p>Using methods in <ref type="bibr">Banks and Olszewski (1997)</ref>, one can use principal components regression to estimate the local intrinsic dimension of an ABM. This approach can inform the analyst about whether the ABM is underparameterized or overparameterized.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.">Conclusions</head><p>We have reviewed ABMs and their inherent statistical challenges as a call to action for the statistical community. ABMs are used in many disciplines, but few statisticians are addressing their theoretical foundations. If we are to stay relevant for many domains, we need to address this research topic.</p><p>ABMs will be used for the foreseeable future. They are easy to program, and domain experts can embed their beliefs in straightforward ways. But as we suggest, those reasons are insufficient. In most cases, researchers do not know how to best fit ABMs to data, or how to make quantified statements of uncertainty about their predictions. And in general, we do not have an agreed upon procedure for validating ABMs.</p><p>Most ABMs do not have tractable likelihood functions. In those cases, statisticians have limited tools, including emulators and ABC. Emulators have been popular for implementing ABMs and other mechanistic process models, but ABC and related methods may be helpful in certain cases <ref type="bibr">(Hooten, Wikle, and Schwob 2020)</ref>.</p></div></body>
		</text>
</TEI>
