Travel demand forecasting is an essential concept within the domain of urban planning, concerned with predicting trip volumes (hereafter referred to as flows) across transportation network links (whether these are streets, rail lines, or airport queues). These predictions enable planners and transportation agencies to anticipate infrastructure needs [6] and mitigate traffic congestion [4]. To make forecasts, planners and engineers have historically relied on observed data from highway loop detectors as well as passively-collected data from smartphones (i.e., GPS traces) [55]. Both sources have two critical limitations: (1) they are prone to sampling bias and (2) they do not provide continuous coverage. Loop detectors generally do not capture non-motorized vehicle trips (i.e., a walk) while also being limited to a subset of high-volume locations. On the other hand, individuals self-select into sharing location-based service (LBS) data, which ends up capturing only a subset of the population (i.e., ones that own smartphones and allow data sharing).
The limitations of observed flow data necessitate Generative Crowd-Flow Models (GCF), a class of travel demand forecasting methods traditionally derived from physical principles based on the associations between distance, the respective sizes of the population, and the propensity to travel [2]. These include the long-standing Gravity model [74], the Radiation model [52], and the Exploration-Preferential Return model [53]. Some of these models have been widely adopted by planning agencies for future travel demand forecasting [8]. We note that the term generative is used in the sense that the models are used to generate synthetic flows (as discussed in [13]), not to estimate the joint distribution between flows and the set of associated predictors.
Recent advances in machine learning (ML) have allowed the creation of more advanced GCF models, capable of producing movements in a city without strict distributional assumptions. Based on neural networks, these methods are known to be more flexible in the types of spatial predictors (i.e., number of amenities, point-of-interest reviews) they can incorporate and have been shown to outperform the traditional models [34,51,62,75]. Leveraging historical data and spatial context, GCF models can generate synthetic flow datasets for diverse urban forms and arbitrary time periods. This can be useful for a variety of applications, including in urban planning (i.e., deciding where to place critical infrastructure [17]) and in public health (i.e., contact tracking the spread of epidemics [40,41]).
However, there is little evidence as to whether these new models can create synthetic traces that result in equitable flow prediction for all parts of society or whether they exacerbate social biases by widening the representation gap. These concerns are pressing given the applications of GCF models in planning for public safety, traffic management, and other critical areas of operation. By incorporating a more equitable representation of under-served groups' travel demands, the framework proposed in this article aims at ensuring that future transportation policies, infrastructure investments, and service improvements are informed by a comprehensive understanding of all community needs.
Fairness of Generative Crowd-Flow Models 10:3
Fairness is increasingly recognized as a critical component of ML systems [7,16,24,26,31,35], but little attention has been given to the applications of city and urban planning-related research that directly relies on spatiotemporal data. In particular, the research community has mainly focused on biases that are created in data collection approaches. For instance, a great body of work has focused on understanding and mitigating biases of cellular data and Call Data Records [46,48,73] focusing on phone-ownership [37,64], while others focused on practices to debias activity-based household travel surveys [5,29]. However, little has been examined in terms of measuring and evaluating the biases that are caused by underlying models. Indeed, understanding fairness by observing a model's behavior in the context that it is intended to be used is crucial for equitable policy planning.
In this work, we introduce a framework for evaluating and comparing the fairness of GCF models, providing multiple possible definitions of fairness to guide the analysis. Through this effort, we (1) bridge the gap between the fairness in ML literature and the human mobility literature and (2) define the first set of fairness metrics for GCF models. Leveraging these metrics, we propose a stochastic biased sampling methodology that generates multiple permutations of an empirical crowd-flow dataset, with each exhibiting varying degrees of intentional bias. This allows us to simulate the underlying data-generating process for empirical flow data, prone to sampling and self-selection bias. Using these perturbed datasets, we measure the impact of biases on the underlying GCF model, allowing us to evaluate and compare model robustness in terms of fairness and accuracy. Thus, the contributions of this work are three-fold:
-we propose definitions of group fairness for GCF models in the context of their application in transportation planning. Such definitions allow planners and practitioners to measure the equity of synthetic datasets derived from generative crowd-flow models; -we develop a framework to decouple data bias from model bias by simulating the datagenerating mechanism of observed crowd flow sets. We release this framework as an opensource repository 1 that can be used by researchers and planners to assess and compare various approaches; -we showcase our experimental results on two datasets and three GCF models, highlighting the tradeoffs between accuracy and fairness;
The rest of the article is structured as follows: In Section 2, we review the literature on GCF models and provide a background on ML fairness and discuss relevant theoretical framings to ground our work based on perspectives in transportation equity. Section 3 posits our methodological approach and introduces metrics that bridge the fairness criteria and equity in transportation planning. Here we also describe our stochastic biased sampling strategy and introduce components of our framework. In Section 4, we describe our experimental setting including datasets we used and baseline models we used to evaluate our framework. In Section 5, we report the results of applying our framework on two datasets and three different models. Finally, we provide concluding thoughts along with implications and limitations of our work in Section 6.
Crowd-flow prediction, also known as Origin-Destination (O-D) estimation, has widespread applications across various domains, each leveraging it for different practical goals. In transportation engineering, OD flows are integral to solving problems such as assessing vulnerability of transport systems, traffic planning, and predicting future transportation demand [9,67]. In urban planning, an understanding of crowd-flows helps inform city layouts by understanding the interaction between different zones, enabling planners to optimize land use and infrastructure [14,25]. Similarly, the domain of public health leverages observed flow data to model the spread of infectious diseases [1,30], as population mobility patterns directly impact transmission dynamics. Additionally, crowd flow datasets have seen applications in regional economics for studying migration patterns and other spatial economic interactions [28,44,45].
The crowd-flow forecasting task can be vulnerable to sampling bias, where data collection methods over-or under-represent certain groups (e.g., smartphone users while excluding cashbased commuters); population bias, where mobility patterns vary across different communities (e.g., urban centers with diverse transit options versus car-dependent suburbs); and socioeconomic inequality bias, where historical disparities in transportation infrastructure lead models to reinforce existing inequities (e.g., underestimating demand in underserved areas, justifying continued neglect in transit planning).
Traditional crowd flow models rely on mathematical approaches to capture the relationship between population mobility and spatial interactions. The gravity model [74], inspired by Newton's law, estimates movement between zones based on population size and distance, with various adaptations such as globally-, production-, and doubly-constrained models. The Intervening Opportunities Model (IOM) [54] modifies the distance-flow relationship by accounting for opportunities encountered between the origin and destination. The Radiation Model [52] offers an alternative, likening population movement to radiation, where decisions are absorbed by regions based on factors like job availability. However, recent works have demonstrated that these approaches may oversimplify the intricate dynamics of human mobility.
Recent advancements in deep learning have led to the adoption of ML models for crowd-flow generation which can leverage large datasets and sophisticated feature engineering (See [13] for a full survey). Deep learning approaches such as Deep Gravity [51] proposed by Simini et al. have emerged as powerful methods for extracting and generating complex spatial dependencies within urban networks. Other approaches such as those based on Graph Neural Networks (GNNs) [49,50], Convolutional Neural Networks (CNNs) [12,71], Generative Adversarial Networks (GANs) [70,72], and Long-Short Term Memory (LSTM) networks [47,65] have transformed mobility flow generation, with neural networks unraveling complex spatial and temporal mobility patterns. At their core, these are black-box models that learn patterns of flow representation from input OD data and are capable of generating complex human mobility patterns, utilizing deep learning to encapsulate the statistical, spatial, and temporal nuances of actual trajectories, aiming to closely align the generated data with real-world patterns. However, as noted by [42], while these algorithms excel at their tasks, their lack of explainability may result in "erosion of trust from users or ethical issues since resulting predictions may perpetuate the biases characterizing the empirical data". To this end, in this article, we aim at creating a set of methods and a framework that can enable the research community to quantify biases that are caused by the underlying GCF models.
The attention around societal concerns of fairness, accountability, transparency, and ethics in ML and data mining has seen a noticeable increase recently [7,16,24,26,31,35]. Literature on fairness in ML tends to focus on the absence of any prejudices or favoritism toward an individual or group based on their inherent or acquired characteristics [36]. The majority of ML fairness research focuses on preventing automated decisions from disproportionately favoring advantaged groups or individuals. The most widely adopted fairness metrics in ML are group-based metrics [15,22], which aim at ensuring independence between a model's predicted outcomes and sensitive attributes such Fairness of Generative Crowd-Flow Models 10:5 as age, gender, and race. Below, we outline some of the commonly used metrics for assessing group fairness in ML models:
-Demographic Parity [19] states that the demographic groups should receive similar decisions, inspired by civil rights laws in different countries [3]. It argues that a disadvantaged and advantaged groups (in terms of the sensitive attributes such as race or gender) should receive similar treatments: -Equalized Odds [22], is a measure of separation criteria in fairness and is satisfied if the predictor's output is independent of the sensitive attribute(s), but still conditional on the ground truth. Equalized odds are measured as the rate of true positive rates (TPR) and false positive rates (FPR) being equal across different groups defined by a sensitive attribute like race or gender, essentially ensuring that the probability of a positive prediction is the same for individuals with the same outcome (positive or negative) regardless of their group membership.
In the context of spatio-temporal data and its applications such as equitable transportation and smart mobility [38], attention has also been mainly devoted to group fairness. Transportation equity heavily employs statistical tests for equity analysis, which is appropriate for discovering unfairness [68,69]. Such metrics are often defined based on census tract level information which offers an aggregate demographic characteristic of the residing population [68].
In transportation contexts, disparities in access to transportation infrastructure can lead to biased mobility predictions. For instance, if a city's public transit system is concentrated in a uent areas while lower-income neighborhoods have fewer transit options, a model trained on historical transit ridership data may reinforce these inequalities by predicting lower demand in underserved areas (due to uneven access), potentially justifying continued neglect in transportation planning.
In the realm of O-D estimation, the concept of equity is particularly underdeveloped, especially when considering representativeness in model outputs. This gap presents an opportunity to bridge philosophical perspectives on equity with the specific challenges inherent in predicting crowd flow, a task that frequently involves diverse populations with varying needs and levels of access. In doing so, we acknowledge that a single view of equity may not fully capture the multidimensional implications of planning decisions on different groups. Table 1 provides a summary of these perspectives, which includes Formal Equality, Sufficientarianism, Rawlsian Justice, al Sadr's Egalitarian Approach, and Procedural Justice. For a comprehensive examination of equity norms in transportation, we refer readers to [33].
Out of these perspectives, we fundamentally focus on Formal Equality, which posits that all individuals should be treated equally, without regard to social or economic circumstances. The reason for this is that formal equality has a straightforward interpretation in the context of empirical flow data-regardless of differences between two locations, all location pairs must have equal representation in the model. In practical terms, achieving formal equality would mean ensuring that the model does not inherently prioritize certain regions or routes over others, aiming instead for uniformly balanced accuracy across all pairs. Moreover, Formal Equity closely relates to the definition of Demographic Parity. Although our focus is on formal equality, in Section 6, we discuss how other perspectives could be implemented and measured as broader implications of our work. In the next section, we discuss how we translate this perspective to a quantifiable metric. 3 Methodological Framework 3.1 Overview Human mobility patterns are inherently complex and often yield biased data due to incomplete observation [20]. To evaluate additional biases introduced by GCF models, it is crucial to replicate data biases through controlled sampling. We propose an intentional-perturbation methodology to assess how GCF models respond to biased input data and evaluate their fairness and predictive accuracy across diverse demographic groups. This approach is motivated by the hypothesis that models relying on complex structures to generate human flows may exacerbate biases present in the underlying data. Figure 1 presents the overview of our open-source frameworkfoot_2 which includes the following key steps. For clarity, a table of all symbols and notation used in this article is provided in Appendix Table A1.
(1) Biased Sampling: We perturb observed data, generating multiple origin-destination-flow samples-each exhibiting different degrees of intentional bias. These biased samples aim at reflecting variations caused by data collection methodologies in the real world. (2) Model Training and Testing: For each biased sample, a GCF model is trained independently.
This step allows us to explore the impact of biased training data on models of a specific type, ensuring that we control for intra-model variation. After training, each model is tested on a standardized, unbiased testing dataset to simulate testing against the real-world ground truth.
(3) Outcome Evaluation: We quantify model performance using two key metrics: fairness and accuracy. Fairness is measured through metrics that relate to the theoretical equity framework discussed in the previous section, and accuracy is defined based on GCF literature as we describe next.
To simulate representativeness bias in flow data, we systematically modify observed flows between an origin and its destinations. The adjustments are governed by either the demographic attributes of the destinations or the disparity in demographic attributes between the origin and destinations.
We base this approach based on theoretical findings in real-world data generating processes-a swath of previous literature explores the consistent correlations between individuals' socioeconomic characteristics and their sampling probability within observed datasets [23,42,66].
Our perturbed sampling approach begins by sorting the destination regions either in ascending or descending order based on two distinct criteria:
-Demographic sampling: Sorting by the demographic attribute of each destination.
-Demographic disparity sampling: Sorting by the absolute difference in the demographic attributes between the origin and each destination.
These methods enable the creation of biased samples across varying demographic profiles. Demographic sampling targets destinations at the extremes of the demographic attribute spectrum, facilitating up-sampling or down-sampling of flows accordingly. Conversely, demographic disparity sampling manipulates flows to destinations with intermediate demographic values, introducing bias in a distinct manner. For example, if the chosen demographic attribute is the Social Vulnerability Index (SVI)ranging from 0 (lowest vulnerability) to 1 (highest vulnerability) [18]-using demographic sampling with an ascending sort would up-sample flows to destinations with higher socioeconomic vulnerability, while a descending sort would up-sample flows to those with lower vulnerability.
Once the destinations are sorted, we compute the Cumulative Percentile (CP) for each destination group j (relative to origin i) to quantify the relative position of the group within the distribution of all destinations. The CP is calculated as follows:
where P k denotes the population of the kth destination group, n is the total number of destination groups for origin i, and j ∈ [1, n] is the index of the current destination group. The numerator j-1 k=1 P k + 0.5P j positions the jth destination group at the midpoint of its population, while the denominator n k=1 P k normalizes the cumulative sum to a percentile scale. This ensures that CP i, j ranges from 0 (for the smallest cumulative population) to 1 (for the total population), providing a consistent measure of the relative standing of destination groups.
The Bias Factor (BF) is derived from CP to capture the impact of the destination group's position on adjusted flow. BF is calculated as
where α is a steepness parameter controlling the shape of the sigmoid curve. The BF assigns weights that transition smoothly around the median CP (0.5), with smaller values of α resulting in gradual transitions and larger values producing sharper changes. Figure 2 illustrates the effect of α on BF i, j , showing smoother transitions at lower values (α = 5) (only emphasizing distinctions between groups at extremities) and sharper changes for higher values (α = 20) (emphasizing distinctions between groups near the median).
The adjusted flow ˆ (j, k) from origin j to destination k incorporates the BF and is calculated as
where ˆ (i, j) and (i, j) denote the adjusted and original flows from origin i to destination j, respectively. This equation adjusts flows (i, j) using BF i, j , ensuring the adjusted flows ˆ (i, j) are biased in a specific direction while preserving the total outflow n k=1 (i, k) from origin i. This approach retains the original structure of flows while incorporating demographic-based bias.
We can also use the calculated BF to create biased samples by instead calculating probabilities of a unit of flow from origin i being assigned to destination j. For the original flow data, this probability P(i, j) is the assigned flow (i, j) divided by total outflow from the origin n k=1 (i, k). We modify it as below:
The expected value of flow from i to j in a perturbed sample using this probability distribution will be the adjusted flow ˆ (i, j), calculated in Equation (3).
Fairness of Generative Crowd-Flow Models 10:9
To evaluate and compare the performance of models under our framework, we assess three key aspects: accuracy, fairness, and robustness. These metrics collectively capture the effectiveness of the model, its equitable treatment across population groups, and its stability under varying conditions.
To measure the accuracy of a given GCF model, we evaluate its fit to the test data. A standard metric in the GCF literature is the Common Parts of Commuters (CPC) [32], which quantifies the overlap between predicted flows and observed flows. We compute CPC as
where (i, j) represents the observed flow from origin i (in the set of origins O) to destination j (in the set of destinations D), ˆ (i, j) denotes the predicted flow from the model, and N denotes the total number of O-D pairs. CPC ranges from 0 to 1, where 1 is a perfect match between the generated flows and the ground truth and 0 is a poor performance with no overlap between generated flows and real flows. CPC is widely used due to its simplicity and interpretability, as it reflects the percentage of correctly located commuting connections. However, it is a directionless metric and does not capture whether errors arise from overestimation or underestimation, both of which have significant implications in real-world applications. Overestimation in certain regions may result in excessive allocation of resources, while underestimation can lead to insufficient support for commuting needs. To address this, we introduce two complementary metrics, Overestimation (OE) and Underestimation (UE):
OE captures the average magnitude of flow overestimation, where positive discrepancies between predicted and observed flows are summed and normalized by the total number of O-D pairs (N ). On the other hand, UE measures the average magnitude of flow underestimation, summing negative discrepancies between predicted and observed flows. Both metrics have the range (0, ∞), where higher values indicate greater over-or underestimation.
Metrics. We define fairness as the extent to which model outcomes differ across geographies characterized by varying demographic distributions. Specifically, we segment O-D pairs into groups based on regional demographic/economic attributes. These sensitive attributes may include socioeconomic factors such as income levels, educational attainment, transportation accessibility, and housing availability.
To calculate disparate treatment among groups, we segment the sensitive attributes of origins and destinations into deciles. We obtain a set of groups as the Cartesian product of the set of origin deciles with destination deciles: G = {G 1 , G 2 , G 3 , . . . ,G 100 }, for a total of 100 groups. Let CPC represent the CPC value for each group . We define the probability distribution over groups as follows:
Per the assumptions of formal equality, perfect fairness corresponds to a uniform target distribution U (G ) = 1 |G | for each group G . Using this, we define an unfairness metric based on the Kullback-Leibler divergence (KL-divergence) between the uniform distribution and the observed probability distribution P(G ). This is expressed as
In essence, this metric quantifies formal equality, ensuring that all O-D pairs have equal representation and accuracy, irrespective of sensitive attributes.
Metrics. Finally, to assess the robustness of a model under the scrutiny of perturbed biased data, we evaluate the variation in its fairness and performance. For a given model, each perturbed data sample yields a point (F , P) in the two-dimensional fairnessperformance space, where F denotes the fairness metric and P represents the performance metric. As we vary the input data through successive perturbations, these points collectively form a polygon whose area characterizes the model's sensitivity. We define the Perturbation Robustness Area (PRA) as the area A of the polygon encompassing all such points. A model that is minimally influenced by input perturbations will exhibit a small PRA, with the points clustering tightly and thus approximating a degenerate polygon (effectively a line segment) in the (F , P)-plane.
We also quantify the model's sensitivity along each axis. Specifically, Fairness Sensitivity is defined as the range of the points along the fairness dimension, and Performance Sensitivity is defined as the range along the performance dimension. Formally, if the set of fairness values from the perturbed samples is {F m } M m=1 and the set of performance values is {P m } M m=1 , then the Fairness Sensitivity is (max m F m -min m F m ), and the Performance Sensitivity is (max m P m -min m P m ).
In this section, we describe baseline models and datasets we used for testing our framework.
4.1.1 Gravity. Gravity model [2] serves as a fundamental framework for predicting and analyzing the movement of people between two locations. Inspired by Newton's law of gravitation, it assumes that the interaction strength between two places increases with their "mass" (typically population size) and decreases as the distance between them grows. Mathematically, this can be expressed as the probability T i j of movement between two locations i and j, represented as
where P i and P j denote the populations of locations i and j, respectively, and r i j is the distance between them. The core assumptions of this model are that the number of trips departing from location i is proportional to its population P i , the appeal of destination j depends on its population P j , and that increasing distance imposes a cost that reduces the likelihood of interaction [43]. This model can be further generalized as
where K is a constant, P i and P j represent the population for the origin and destination, respectively, and f (r i j ), known as the deterrence function, is a decreasing function of distance, commonly
Fairness of Generative Crowd-Flow Models 10:11 represented by either a power law or an exponential decay function [43]. Parameters σ and β allow the model to be fine-tuned, capturing how changes in population or attractiveness factors at origins and destinations impact the overall flow. Gravity model can be further refined to more accurately represent real-world movement patterns by introducing constraints, such as fixing total flows from origins or to destinations, and incorporating additional variables like socioeconomic factors. Despite being simple and easy to interpret, the gravity model has notable shortcomings. It struggles to fully capture complex movement patterns and often underestimates the variability present in real-world mobility data.
Gravity. Deep Gravity model [51] is an extension of constraint gravity model that is based on a nonlinear variant of the multinomial logistic regression obtained by adding hidden layers, which introduce nonlinearities and build more complex representations of the input geographic features such as land use areas, road network features, transport facilities, and so on. Unlike the classic Gravity model, which relies primarily on population and distance, the Deep Gravity model incorporates detailed geographical information extracted from Open Street Maps (OSM). Deep Gravity model significantly improves flow generation over traditional gravity models, particularly in densely populated regions.
Non-Linear Gravity model is a variant of Deep Gravity model where the neural network structure remains constant, but it does not incorporate embedded features. This approach allows the model to focus on capturing non-linear relationships directly between inputs like population and distance, rather than leveraging additional feature embeddings.
Our study utilized two datasets, one for Washington State and one for New York State, both aggregated at the census tract level. The flow file in each dataset captures population movement between O-D pairs, with associated GeoIDs and flow counts to train and test models on.
The flow data for both the Washington and New York datasets were sourced from the University of Wisconsin-Madison's GeoDS Lab [27] who, in conjunction with SafeGraph, have collected a dataset which captures human movement patterns across the U.S. from January 2019 to April 2021.
This data is based on mobile device locations from over 100 million devices and provides information on mobility trends aggregated to the census tract level and processed into daily and weekly flows. Each data entry includes an O-D pair with the timeframe the data was collected over, GeoIDs and coordinates for the origin and destination, and two flow values. The first, visitor flow, is the raw number of devices traveling between the two locations observed by SafeGraph. The second, population flow, is a scaled version of visitor flow that accounts for the underlying population count and the proportion of the population captured via mobile devices. The flow data used in each of our datasets were the weekly population flow values from June 10th to 16th, 2019, collected for all census tracts in Washington and New York states using the download_weekly_data script provided in the repository by [27].
Point of Interest ( POI) Data. As some of the baseline models require contextual information about the city, we collected POI data from OSM. Using this data we create a feature file, which serves as input for model training and testing and is primarily derived from the U.S. Census Bureau. Specifically, it uses population counts from the Decennial Census (DEC) redistricting data for 2020 in Washington [60] and New York [61]. And while the Gravity and Non-Linear Gravity models utilized only this population data for their feature set, as previously mentioned, Deep Gravity also utilized data from OSM to make use of its deep neural architecture. The OSM data was initially retrieved using the OSMnx packagefoot_3 at the county level for both states. To manage the large volume of geospatial data and map it to the tract level, each county was divided into smaller grid cells using county boundary data from the Washington Department of Natural Resources [63] and the New York State GIS Clearinghouse [39], respectively. Additional details of our preprocessing steps for POI data are included in the Appendix B.
We leveraged sociodemographic-related data from the Centers for Disease Control and Prevention's (CDC) SVI [18]. The SVI is a metric which attempts to quantify the resilience of communities to external stresses on human health, such as epidemics or natural disasters. It is calculated at both the census tract and county levels and consists of four themes: Socioeconomic Status, Household Characteristics, Racial and Ethnic Minority Status, and Housing Type and Transportation. These themes are themselves calculated using 16 U.S. Census variables from the 5-year American Community Survey (ACS). The data we used were the 2020 SVI values for Washington [10] and New York [11], visualizations of each being found in Figure 3. Each SVI dataset was retrieved from the CDC's portal by selecting the corresponding state and year. For our study, we only utilized the values associated with the RPL_THEMES attribute which gives the SVI in the form of percentile rankings, meaning our SVI values range from 0 to 1 with 1 indicating increased vulnerability.
Our study also incorporated geographic data in GeoJSON format to define census tract tessellations and the boundaries of the study regions. These data were sourced directly from the U.S. Census Bureau's TIGER/Line Shapefiles portal [57], which provides standardized geographic datasets. The New York tessellation data was retrieved from the Census Bureau's tract level files in the New York repository [56], and the Washington tessellation data from its Washington repository [59]. The boundary files, delineating the broader study areas, were also drawn from the TIGER/Line repository. Specifically, the Public Use Microdata Areas (PUMAs) files for regional delineation [58].
Preprocessing. When preprocessing our data, we employed a consistent methodology for both datasets. The process began with the retention of all OD flow entries from [27], encompassing each state's dataset for the week of June 10th, 2019. From there we began to integrate the data from our other described sources and found that 306 of the GeoIDs from the Washington flow data and Fairness of Generative Crowd-Flow Models 10:13 553 of the GeoIDs from the New York flow data did not align with the GeoIDs from the 2020 DEC for each state. We suspect this is due to changes in geographic boundary definitions or updates to census tract identifiers between the collection of the SafeGraph mobility data and the publication of the 2020 DEC. However, the SafeGraph mobility data includes latitude and longitude for each tract's centroid, when combined with Census Bureau Shapefiles for each state, this information enables us to map each tract to its 2020 DEC counterpart and align them using DEC GeoIDs as unique identifiers.
During this alignment process, we identified 4 null SVI values in the CDC's Washington data and 134 null values in the New York data. Additionally, we observed that the New York flow data contained 17 tracts absent from the CDC's SVI dataset. While initially concerning, these discrepancies are consistent with known limitations in SVI data collection and calculation. Specifically, the CDC excludes all census tracts with a total population of zero, explaining the extra tracts present in the New York flow data. Additionally, tracts with incomplete data from the ACS are often assigned null values in the SVI dataset. This occurs when some demographic or social variables required for calculating SVI rankings are unavailable or cannot be derived due to insufficient sample sizes or reporting issues. To handle these missing values, we used the SVI value for the county in which each tract is located would provide the most consistent approach.
Train Test Split. Similar to the methodology utilized in [51], we have defined a uniform dataset format that divides geographical spaces into 25km * 25km tiles, with each square tile potentially encompassing multiple GeoIDs. However, larger GeoIDs may span multiple tiles, ensuring finer spatial resolution and preserving important geographic detail. The train-test split was performed randomly at the tile level. Importantly, we excluded any GeoIDs that appeared in both the training and test datasets to avoid issues caused by cross-train-test contamination, which could lead to missing data in either set and potentially affect model validity.
After this split, we obtained distinct sets of tile IDs for the training and test datasets. The training dataset was subsequently subjected to a biased sampling process, resulting in different versions of the training set, each reflecting a distinct bias influenced by various demographic or geographic factors. The test dataset, on the other hand, remained untouched throughout the entire process and was used exclusively in the final evaluation phase to ensure an unbiased assessment of model performance. We recognize that the test dataset itself may still carry inherent biases. To account for this, we supplement accuracy-based evaluations with fairness metrics, ensuring that model performance is assessed not just in terms of predictive accuracy but also in its ability to provide equitable forecasts across different groups.
We utilize the biased samples generated in Section 3.2 to train the baseline models. Each model is trained separately on each individual sample, and this process is repeated for all samples across all models. To ensure comparability, all hyper-parameters remain consistent for the same model type across different biased samples. This process produces multiple versions of each model type, with each version being trained using a specific biased sample. Thus, for every biased sample, we generate a unique instance of the model, allowing us to analyze how different types of biases affect model performance. Appendix C reports our full list of hyper-parameters.
We first analyze the results of unbiased NY dataset in Non-linear Gravity model (α = 5) before discussing the generalized findings across other perturbed samples and comparing with other models. Figure 4 presents the detailed distribution of the CPC values in unbiased sample for assorted O-D SVI buckets. In an ideal world with no biases in the model, we would expect a heatmap of uniform gradient, indicating the model is capable of reconstructing flows for each O-D pair equally. The darker the color, the higher the CPC value for each O-D pair, reflecting a generation of flow data that matches the ground truth flow accurately.
We can observe that the bottom left of the heatmap which corresponds to the origins and destinations with lower SVI (i.e., flow from least vulnerable areas to least vulnerable areas), are darker, thus exhibiting higher CPC values. In contrast, the flows between the least well-off economic areas (i.e., top right corner) suffer from the lowest CPC values, indicating that the Non-linear Gravity model trained by the unbiased dataset is not fair.
Next, we measure the impact of other perturbed bias datasets on the Non-linear Gravity model. Figure 5 illustrates the results of this experiment, where each data point is the result of an unbiased or perturbed data sample. We can observe that the unbiased sample (i.e., original Non-linear Gravity performance), denoted by the red circle, achieves a mean CPC of 0.28, which summarizes the average flow reconstruction accuracy across all O-D pairs as also shown in the heatmaps (Figure 4). The results of unbiased sample act as a benchmark for evaluating the model on other perturbed data samples. Experiments related to sampled perturbed data are denoted by square for demographic disparity sampling method and triangle for demographic disparity sampling method. Blue points denote ascending approach and gray points denote descending.
Notably, in this experiment, the unbiased sample is positioned slightly below the center of the hull, balancing performance (mean CPC) and unfairness (KL-divergence). Compared to the unbiased sample, Non-linear Gravity models trained on the perturbed samples using ascending sorting (hereafter referred to as ascending outcome) exhibit lower performance but achieve greater fairness. This is expected as ascending sorting aims at increasing sampling opportunities for the more vulnerable groups. Conversely, those models trained on the perturbed samples using descending sorting (hereafter referred to as descending outcome) sacrifice fairness to attain better performance, as the descending sorting exacerbates biases related to most vulnerable group (i.e., reduces their representation in the sample data).
The areas enclosed by ascending outcome and the unbiased sample and by the descending outcome and the unbiased sample, are represented by a solid line and a dotted line, respectively. A Fairness of Generative Crowd-Flow Models 10:15 clear distinction can be observed: the area enclosed by ascending outcome is significantly larger than that of the descending outcome, indicating that the Non-linear Gravity model exhibits greater variability and is more sensitive to perturbations caused by ascending sorting; while the model is less prone to variation induced using descending sorting. In our further evaluation of the robustness and sensitivity of different models, we use the average area, height, and width of the hulls formed by ascending and descending outcomes to represent each model.
For each sample (point) in Figure 5, there is a corresponding heatmap representing the performance distribution across O-D demographic buckets. Figure 6 includes heatmaps for four different types of samples, for these experiments. Generally, a consistent gradient emerges in all heatmaps, where CPC tends to decrease from the bottom-left corner (i.e., lower SVI O-D pairs), toward the top-right corner (i.e., higher SVI O-D pairs). Compared to the heatmap of the unbiased sample (Figure 4), in the ascending outcome, we can observe that the diagonal region (bottom-left to topright), which originally exhibited very high CPC values in, becomes diffused outward to the surrounding areas to a certain extent. In contrast, the descending outcome intensifies the diagonal line's color, further concentrating CPC along the region. The difference becomes even more pronounced when α = 20 (i.e., increased perturbation magnitude).
In this section, we analyze the experiment results for different models on different performance metrics from the perspectives of robustness and sensitivity, as shown in Figure 7 and Table 2. The evaluation provides critical insights into the performance across various metrics, focusing on CPC, OE, and UE when different values of α are applied.
It is important to note that meaningful comparisons of robustness and sensitivity can only be made between models when evaluated under the same performance metrics and that during the perturbation sampling process, the random seed influences the probabilities, which can affect sensitivity measurements due to some outliers.
Analysis. Robustness analysis highlights the dynamics of model stability, which is quantified by the area occupied by each model on the plot, as calculated using the mean value of the ascending and descending outcome hulls. A smaller PRA corresponds to smaller Fairness of Generative Crowd-Flow Models 10:17 The bold values denote the lowest variation in the metric for each given α across models.
variability, indicating greater stability and robustness. The Gravity model consistently exhibits the smallest PRA across different performance metrics. Therefore, we conclude that the Gravity model demonstrates the highest robustness relative to the other models, which means it is least susceptible to fluctuations and biases in input data and makes it a reliable choice in settings where consistency is critical. This finding aligns with the long-standing understanding that more complex models are more likely to overfit to their training data-Gravity model, being a simpler and more parsimonious approach, benefits from a reduced risk of overfitting, allowing it to generalize more effectively to biased datasets. In contrast, the Deep Gravity model and the Non-linear Gravity model display larger variability in robustness under different performance metrics. Specifically, when α = 5 , the PRA of Deep Gravity model is significantly smaller (PRA = 0.0495) than that of Non-linear Gravity (PRA = 0.1011) model in terms of OE. However, in the CPC and UE metric, the robustness of Deep Gravity model and Non-liner Gravity model are close to each other. A similar pattern emerges when α = 20. These observations suggest that the variability in robustness for Deep Gravity model and Non-linear Gravity model depends on the specific performance metric being evaluated. When only comparing Deep Gravity model and Non-linear Gravity model, we observe that the PRA of Deep Gravity model is almost always lower than or comparable to that of Non-linear Gravity model, indicating that Deep Gravity model maintains a consistent level of robustness across various perturbations. A key reason for this stability is that Deep Gravity model effectively leverages the POI features-which remain unperturbedregardless of how the flow data is perturbed. As a result, Deep Gravity model demonstrates greater robustness.
Analysis. Sensitivity analysis examines how the models respond to variations in fairness and performance. Sensitivity is quantified through changes in unfairness value (i.e., the height of the hull) and performance value (i.e., the width of the hull). Consistent with the robustness analysis, the calculation is based on the mean values derived from the ascending and descending outcome hulls. Larger sensitivity indicates greater variation in the specific direction.
Fairness Sensitivity. Overall, the Gravity model demonstrates the smallest fairness sensitivity, particularly under the UE metric. Notably, under the OE metric, the Gravity model exhibits a much larger fairness sensitivity. Among the three models, the shape of Gravity model's hull is extremely elongated and is almost parallel to the y-axis (unfairness values) indicating substantial variations in fairness at similar performance levels. The samples generated by ascending sorting are the primary contributors to this increased fairness sensitivity in Gravity model. In the CPC metric, despite the similar robustness levels of the Deep Gravity and Non-Linear Gravity models, Deep Gravity model's fairness sensitivity is nearly twice that of on-Linear Gravity model, highlighting Deep Gravity model's greater susceptibility to fairness-related variability.
Performance Sensitivity. A significant finding is that the Gravity model consistently shows minimal performance sensitivity under both OE and UE metrics, compared to the other two models. This can be attributed to its simpler structure, which is less likely to be influenced by sampling perturbations. However, in the CPC metric, the performance sensitivity of Gravity model is slightly higher than Deep Gravity model, which indicates that Gravity model is more likely to sacrifice fairness in favor of improved CPC performance.
The α significantly influences model performance, with distinct effects observed at α of 5 and 20. When α = 5, the models demonstrate higher stability and constrained variability, while α = 20 amplifies robustness and sensitivity, leading to larger variability across metrics.
For the more robust and less sensitive model, Gravity model, the amplifying effect of α is less pronounced. However, an exception arises in the OE metric, where α = 20 increases fairness sensitivity to more than 2.5 times that observed at α = 5. In comparison, the less robust and more sensitive models, Deep Gravity model and Non-Linear Gravity model, exhibit much stronger responses to changes in α.
The amplifying effect of α is neither proportional nor uniform across different metrics. For instance, in the CPC metric, α has a more pronounced impact on performance sensitivity for the Non-Linear Gravity model; and for the Deep Gravity model, while α leads to increases in both fairness and performance sensitivity, its robustness remains nearly unchanged.
Figure 8 shows the evaluation of different models on WA dataset under the CPC metric when α = 5 and Table 3 presents the results of different models under different performance metrics with α of 5 and 20. Consistent with the findings from NY dataset, the Gravity model exhibits the smallest PRA and shows the lowest variation in sensitivities, indicating its superior stability across fairness-performance tradeoffs.
Interestingly, the Deep Gravity model achieves both the best performance and the highest level of fairness in this evaluation. The result may stem from the significantly smaller size of WA flow data compared to NY flow data, which is likely to result in lower complexity or less noise. It further enhances the generalization ability of the Deep Gravity model as it allows the POI features incorporated into it to more effectively capture the patterns and structure of the data, thereby optimizing both performance and fairness simultaneously. Appendix D provides the detailed results for the WA dataset.
In this section, we discuss the main implications of our work and the limitations that it carries, as well as opportunities for future work. Table 3. WA Results of Different Models on Different Performance Metrics with α of 5 and 20 Performance Metric Model Perturbation Robustness Area (PRA) Fairness Sensitivity (Height) Performance Sensitivity (Width) α = 5 α = 20 α = 5 α = 20 α = 5 α = 20 CPC Gravity 3.2752e-6 1.0633e-5 0.0007 0.0015 0.0418 0.0805 Deep Gravity 6.9343e-5 0.0001 0.0016 0.0038 0.0717 0.1090 Non-Linear Gravity 0.0002 0.0003 0.0046 0.0072 0.0831 0.1016 OE Gravity 0.0008 0.0031 0.0039 0.0040 1.1706 2.2595 Deep Gravity 1.0596 1.4500 0.1206 0.1171 23.5287 36.2521 Non-Linear Gravity 0.3286 0.9831 0.0347 0.0920 27.4523 33.3416 UE Gravity 2.8129e-5 0.0002 0.0005 0.0006 1.2091 2.3230 Deep Gravity 0.7003 0.5745 0.0764 0.0724 24.9122 37.6530 Non-Linear Gravity 0.5101 0.9140 0.0336 0.0641 28.6266 35.0558
The bold values denote the lowest variation in the metric for each given α across models.
Sustainable Development Goals. 4 The findings of our study have significant practical implications for urban planning and policy-making. By proposing and applying a novel set of fairness metrics to GCF models, our research aids in critically evaluating these models to ensure they do not perpetuate or exacerbate existing social inequalities. This is particularly crucial in urban planning, where biased mobility predictions can lead to unequal resource allocation and infrastructural development. Urban planners can use the insights from our fairness benchmarks to guide the deployment of GCF models, ensuring that the infrastructure developments and service improvements they plan reflect the needs of all segments of the community, including traditionally under-served groups.
Our work also extends to critical areas such as public safety and traffic management, where equitable flow predictions are essential. The improved understanding of bias within GCF models can help policymakers allocate resources in a way that equitably addresses the mobility needs of all populations, potentially leading to safer and more efficient urban environments.
6.1.2 Theoretical. On a theoretical level, our research contributes to the ML and urban mobility literature by addressing a significant gap: the assessment of fairness in GCF models. Prior to this work, the interaction between model-generated biases and urban planning outcomes had not been thoroughly explored. Our introduction of specific fairness metrics for GCF models enriches the academic discussion, offering tools that can be used to dissect and understand the nuances of biases within these systems. This encourages a critical reassessment of how predictive models are developed and evaluated in light of equity considerations, urging the research community to prioritize fairness in algorithmic design and application.
Furthermore, our open-source framework for measuring and benchmarking fairness in GCF models promises to foster further research to mitigate biases in various domains of predictive modeling. This could lead to broader applications and refinements of our proposed metrics, enhancing the reliability and ethical grounding of predictive analytics in urban planning and beyond.
We acknowledge that equity is not a single perspective domain and while we have done our best in scoping this work and have introduced a novel metric to measure fairness based on the perspective of the Formal Equality, other perspectives are yet to be examined. To encourage future contribution to this research, here we put forward alternative ways that other equity perspectives could be framed in the context of GCF evaluation (first shown in Table 1).
-Sufficientarianism: Focuses on meeting a minimum standard or threshold, ensuring that every individual or group attains a "sufficient" level of advantage or resource. Applied to crowd flow models, this principle would demand that predictions for each O-D pair meet a minimum accuracy standard. If the model's predictions for certain pairs fall below this threshold, adjustments would be warranted to ensure that all groups are adequately represented and not disproportionately impacted by low-accuracy predictions. In our framework, this can be measured by simply putting a threshold that relates to the minimum accuracy and then computing the count of O-D pairs above threshold divided by total number of O-D pairs. Such metric when closer to 1 would present sufficientarianism and when closer to 0 will indicate need for intervention. -Rawlsian Justice: Accepts that inequalities can be justified if they ultimately benefit the least advantaged members of society. In the context of crowd flow prediction, a Rawlsian approach would prioritize accuracy for groups or regions that are typically marginalized or under-served. Here, model evaluation metrics would impose higher penalties for inaccuracies affecting these vulnerable groups, recognizing the greater harm that inaccuracies in data-driven decisions may impose on those with limited mobility options or resources. -Al-Sadr's Egalitarian Approach: Emphasizes improvements in conditions specifically for disadvantaged or impoverished groups. For crowd flow prediction models, this approach would dictate a focus on optimizing predictions in ways that benefit low-income or underserved areas. Unlike Rawlsian justice, which seeks mutual benefit across society, al-Sadr's approach directs resources and model improvements toward the poorest groups, aiming to uplift these communities more directly. For Both Al-sadr and Rawlsian Justice, we envision that approaches that integrate in-process de-biasing techniques, where the focus is on optimizing the performance of the under-served areas by applying regularizers to the model's loss function can attain fairness that corresponds to this frame of ethical thinking. -Procedural Justice: Emphasizes fairness in the processes used to reach decisions, asserting that equitable outcomes are the result of fair and transparent procedures. Applied to
Fairness of Generative Crowd-Flow Models 10:21 crowd flow models, procedural justice would require transparency in data collection methods, model training processes, and the handling of potential biases. This perspective ensures that if the processes behind model development are unbiased and inclusive, then the model's outputs can be considered fair. It also suggests that stakeholders should be involved in model development, with feedback loops to address disparities and reinforce trust.
Other limitations of our framework include the assumption that the POI data from OSM is accurate and recent representation of the amenities and that this representation is equal across regions of the city. We note that previous research has indicated that there exists a digital divide on crowdsourced maps and we believe this bias in map representation could also be an indicator of why Deep Gravity model's performance is bounded even under most perturb data.
While our fairness benchmarks provide valuable insights into mitigating bias in GCF models, our approach does not fully account for network correlation effects that may influence inter-group fairness. In real-world mobility networks, travel demand across different groups is interconnected through shared infrastructure, meaning that changes in one group's mobility patterns can indirectly impact others. For example, if multiple communities rely on a single road, an increase in travel demand from one group can alter commute times and congestion levels, subsequently affecting the mobility experiences of other groups. This network-level interdependence poses challenges to traditional fairness definitions, which often assume independent group dynamics. While our study primarily focuses on assessing fairness at the group level, future work should explore fairness metrics that incorporate these network effects to provide a more comprehensive evaluation of fairness in crowd-flow forecasting.
Finally, another limitation of our work is on the perturbation sampling strategy. There are many alternative ways to create the biased samples, for example, pink noise sampling where the sampling is based on the impact of distance on flow [21]. In this study, we opted in for focusing on sampling purely based on the sensitive attributes that distinguishe regions from each other. We envision that a combination of other sampling strategies and ours can be implemented as part of our open-source framework in the future. Additionally, a more systematic data attribution exercise could be incorporated as future work to better analyze the sources and representativeness of our datasets. This would help assess how data collection biases influence model predictions and fairness outcomes, providing a more transparent evaluation of potential disparities in mobility forecasting.
Crowd-Flow Models 10:25 [75] Dongcheng Zou, Senzhang Wang, Xuefeng Li, Hao Peng, Yuandong Wang, Chunyang Liu, Kehua Sheng, and Bo Zhang. 2024. Multispans: A multi-range spatial-temporal transformer network for traffic forecast via structural entropy optimization. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining. 1032-1041. Appendices A Table of Notations Table A1. Table of Notations Symbol Explanation Indices: i, j Index for origin and destination k Index for destinations from a given origin Index for demographic groups in G m Index for perturbed samples Parameters and Calculated functions: α Steepness parameter for Bias Factor CP i, j Cumulative Percentile ranking for destination j from origin i BF i, j Bias Factor for destination j, given origin i CPC Common Parts of Commuters OE Overestimation U E Underestimation D K L Kullback-Leibler divergence Original flow from i to j ˆ ( i, j)
Adjusted flow from i to j ˆ (i, j) Generated flow from i to j
The alignment process involved clipping roads and intersecting land classification zones with each census tract to accurately associate only the OSM data within each tract to it is corresponding GeoID. This process was repeated for both datasets and utilized the same 18 features from the OSM data, These 18 features were calculated to capture critical aspects of land use, transportation infrastructure, and POI for each census tract. A detailed overview of the features and their calculation methods is provided below:
-Land Use Areas (km ):
-residential_landuse: Total area where the landuse key has the value residential.
-commercial_landuse: Total area where the landuse key has the value commercial.
-industrial_landuse: Total area where the landuse key has the value industrial.
-retail_landuse: Total area where the landuse key has the value retail.
-natural_landuse: Total area where the natural key is present. -Road Network (km):
-road_residential: Total length of ways where the highway key has the value residential.
-road_primary: Combined length of ways where the highway key has the value primary or trunk. -road_other: Combined length of ways where the highway key has the value secondary, tertiary, or unclassified. -Amenities:
-transport_point: Count of nodes where the public_transport key has the value stop_position or the amenity key has the value parking. -transport_area (km²): Total area of features with the above transport-related tags.
-food_point: Count of nodes where the amenity key has the value bar, pub, cafe, or restaurant.
-food_area (km²): Total area of features with the above food-related tags.
-health_point: Count of nodes where the amenity key has the value clinic, hospital, or pharmacy. -health_area (km²): Total area of features with the above health-related tags.
-education_point: Count of nodes where the amenity key has the value college, school, kindergarten, or university. -education_area (km²): Total area of features with the above education-related tags.
-retail_point: Count of nodes where the shop key is present.
-retail_area (km²): Total area of features where the shop key is present.
Table A2 shows the hyper-parameters we used for training Deep Gravity model and Non-linear Gravity model.
Table A2. Deep Gravity and Non-linear Gravity Models Hyper-parameters Parameter Description Value Batch Size The batch size of the train and test data. We used the default one in Deep Gravity article [51]. 1 Epochs Total epochs for training the model. 20 Learning Rate Learning rate for each iteration of optimizing the model. 5e-6 Momentum
The momentum parameter for Stochastic Gradient Descent with momentum. 0.9
Figure A1 shows WA evaluation results on different models under different performance metrics with α of 5 and 20.
ACM J. Comput. Sustain. Soc., Vol. 3, No. 2, Article 10. Publication date: April 2025.
https://github.com/DSSG24-CF-Fairness/crowdflow-fairness
https://github.com/DSSG24-CF-Fairness/crowdflow-fairness
https://geoffboeing.com/publications/osmnx-paper/. Last accessed 03-24-2025.
https://sdgs.un.org/goals