It is important to accurately predict pedestrian trajectories [1][2][3]. For example, in situations like crosswalks and crowded public areas, accurately predicting pedestrian trajectories can improve safety and prevent potential accidents [4][5][6][7][8].
In monitoring systems, predicting pedestrian trajectories is pivotal in facilitating the detection of anomalous behaviors [9][10][11]. Additionally, it can help optimize the planning of transportation systems with better insights into pedestrian flow and behavior modeling [12][13][14][15][16].
Forecasting the trajectory of a pedestrian still remains a significant challenge, primarily for two reasons: (1) the complexity of interactions among pedestrians in a given environment and (2) the heterogeneity in individual behavioral preferences. Regarding the first reason, there are multiple factors influencing a pedestrian's trajectory, e.g., static obstacles like trees and roads and dynamic components including vehicles and other pedestrians. As reported by [17], up to 70% of pedestrians in a crowd move in groups, such as families or friends walking together. Such interactions are mainly driven by "social interactions" [18,19]. Regarding the second reason that this remains challenging, different individuals usually display varied behaviors under similar circumstances [20], which makes it complicated to establish a universal behavioral model that fully represents the Physics-based models One classical approach to tackle the challenge of pedestrian trajectory prediction is to utilize physics-based models. The physics-based models are usually characterized by some basic rules or generic functions, taking into account both physical constraints and pedestrians' social or psychological factors [2,[36][37][38][39][40]. One well-known physics-based model is the "cellular automaton model" [41,42]. The cellular automaton model is a discrete model, which is based on the discrete motions of pedestrians traversing a grid of cells, and it is assumed that each cell is in a finite number of states.
The second type of physics-based models is called the "social force model" [18]. The social force model is a microscopic continuous model, which studies the motions of pedestrians by some social forces, such as the destination choice and the need to avoid collisions with other pedestrians. Basically, the original social force model assumes that most of scenarios encountered by pedestrians are standard, such that behavioral strategies acquired through experience can be utilized.
Later, ref. [19] introduced "Nomad", a generalized version of the social force model, which incorporates behavioral rules, and continuous route choice. This activity-based approach allows pedestrians to adapt their movements based on different traffic conditions, e.g., distance to a destination. In [36], the authors provided a comprehensive review of crowd motion simulation models, explaining their characteristics, applicability, and the underlying crowd movement phenomena. Interested in pedestrian behaviors at signalized crosswalks, ref. [43] adapted the social force model and calibrated its parameters using maximum likelihood estimation.
Another type of physics-based model is represented by the category of "velocity-based models", which have been widely used in the game industry and robots [44][45][46][47]. Technically speaking, velocity-based models rely on differential equations, and their associated speed functions depend on the relative positions of the neighboring pedestrians and obstacles. For example, the reciprocal velocity obstacle model [45] is able to navigate multiple agents in real time and generates safe and oscillation-free motions. In [48], the authors proposed the optimal reciprocal collision avoidance (ORCA) model, which can provide local collision-free motions for a large number of agents within a time interval.
Deep learning-based trajectory prediction Inspired by the success of deep learning models [22, [49][50][51][52][53], numerous research studies have focused on utilizing deep learning models for the task of pedestrian trajectory prediction. Social long short-term memory (Social LSTM) [27] is one of them. The Social LSTM model employs a type of recurrent network to learn the sequential movement of each pedestrian. To predict the trajectory afterwards, the "social pooling" mechanism is utilized to aggregate the output of the RNNs. Specifically, the model pools the neighbor hidden states of a pedestrian within a distance threshold. Later, based on LSTMs and Generative Adversarial Networks [30], Social GAN [34] was proposed. Social GAN designed a novel pooling mechanism that calculates interactions according to the relative distances among pedestrians.
Subsequent works have built upon Social LSTM and Social GAN [28,29,35,54]. For example, State-Refinement LSTM (SR-LSTM) [29] proposed a new pooling mechanism, which leverages the intentions of neighboring pedestrians. This approach iteratively refines the states of all pedestrians using a mechanism known as "social-aware information selection". Peek Into The Future (PITF) [28] incorporates visual features and proposes modules that take pedestrian-scene interactions into consideration. The Sophie framework [35] is an LSTM-based generative adversarial network. It utilizes convolutional neural networks (CNNs) to extract scene features, followed by a dual attention mechanism. Subsequently, Sophie combines the attention outputs with the scene features.
Given that graph structures are able to explicitly represent the interactions of pedestrians, there has been a growing research interest in graph-based approaches [55,56]. Graph attention networks (GATs) [55] leverage the architecture of Bicycle-GAN and capture the pedestrian interactions with the help of the graph attention mechanism [55]. Recursive Social Behavior Graphs [56] utilize graph convolution networks, combined with additional social information from expert sociologists.
To directly utilize spatial and temporal information together, ref. [57] proposed the Social Spatial-Temporal Graph Convolutional Neural Network (Social-STGCNN). Social-STGCNN models pedestrian trajectories as a spatial-temporal graph, where the spatial edges represent social interactions between pedestrians, weighted by their relative distances. However, the graph kernel function in Social-STGCNN is still based on some predefined rules, e.g., pedestrians with shorter distances have higher weights. By incorporating the spatial-temporal attention mechanism, we are able to move beyond such "predefined rules". Our proposed model learns to assign varying levels of importances to different neighbor nodes based on their features, while also taking the predefined rules into consideration.
In summary, while most previous works focus on modeling pedestrian interactions, the future trajectory of a pedestrian is usually uncertain and different pedestrians may exhibit distinct preferences regarding their behaviors. Therefore, it is not only crucial to model pedestrian interactions, but also imperative to explicitly quantify the inherent heterogeneity present within pedestrian trajectories. To bridge this gap, we introduce the InfoSTGCAN framework in this study. The proposed framework encapsulates pedestrian trajectories within a spatial-temporal graph, leveraging both convolutional and attention mechanisms across spatial and temporal dimensions. Furthermore, we model the behavioral patterns of each pedestrian as a latent distribution derived from their trajectories. This approach intuitively assigns unique latent codes to pedestrians, corresponding with distinct trajectory styles. Figure 1 illustrates such concept and Table 1 summarizes the differences between the proposed framework with the previous methods. Convolutional graph neural networks Basically, there are two major types of convolutional graph neural networks: spectral-based approaches and spatial-based approaches. Spectral-based approaches develop convolution operations based on the graph Fourier transform, e.g., GCNs [60],and ChebNet [62]. Spatial-based approaches perform convolution directly on the edges, making them suitable for asymmetric adjacency matrices, e.g., GraphSage [63] and DGCNN [64]. To deal with the spatial-temporal data, ST-GCN [70] expands the spatial GCN into a spatial-temporal version for the task of skeleton-based action recognition, incorporating information from a localized spatial-temporal context.
Graph attention networks Attention has been widely used in a series of tasks, e.g., machine translation [71], entity resolution [72] and so on. Proposed by [73], graph attention networks bring attention mechanisms to graph neural networks, which calculate the relative weights between two connected nodes by the attention scores. Later, [74] proposed GeniePath, which controls the flow of information by some LSTM-style gating mechanisms.
Specifically, the main contributions of this paper are highlighted as follows:
1.
We formulate the task of pedestrian trajectory prediction as a spatial-temporal graph and propose a novel trajectory prediction model, InfoSTGCAN. This model takes both pedestrian interactions and heterogeneous behavior choice modeling into consideration. Through a comprehensive list of experiments, we demonstrate the superiority of InfoSTGCAN in comparison to existing baseline methods. 2.
Our proposed method integrates spatial-temporal graph convolution and spatialtemporal graph attention. This fusion enables our method to more effectively model pedestrian interactions by evaluating pedestrian importance using a combination of prior knowledge and data-driven features.
Based on the technique of variational mutual information maximization, our model generates an individual-level latent code for each pedestrian. These distinct latent codes facilitate the generation of trajectories with heterogeneous behavior choices.
The remainder of this paper is organized as follows: Section 2 presents the preliminaries and major notations and defines the problem of human trajectory prediction. In Section 3, we explicate the proposed method, focusing on the spatial-temporal graph network, the variational mutual information maximization, and the multi-objective loss function. The proposed model is then evaluated in Section 4, and the design is validated through a process that includes performance comparison, results visualization, and ablation studies. Finally, we draw conclusions and discuss future research directions in Section 5.
Suppose there are N pedestrians within a scene. Given their past observed trajectories tr 1:N obs during a period of time T obs , our objective is to predict their future trajectories tr 1:N pred over the forthcoming time period T pred . In this study, we jointly predict the future trajectories of all pedestrians.
To clarify, we begin by defining the observed trajectory of a pedestrian n (n ∈ {1, . . . , N}). Specifically, for pedestrian n, its observed trajectory tr n obs can be formulated as follows:
where (x n t , y n t ) are a pair of random variables that indicate the location distribution of the n th pedestrian at time step t. Following a similar formulation in [27,57], the probability to predict an individual-level latent code for each pedestrian that possesses high mutual information with the future predicted trajectory. As a result, it enables the model to effectively capture the complexity and inherent heterogeneity of the pedestrian trajectories.
To summarize, the proposed approach is not only able to present a holistic view of their interactions but also able to capture the heterogeneity in pedestrian movements. Details regarding the optimization of the proposed method, including the multi-objective loss function, can be found in Section 3.3.
To begin with, we need to construct the spatial-temporal graph representation from pedestrian trajectories. Essentially, a spatial-temporal graph is an attributed graph where the node attributes evolve through time [59]. The key idea behind the spatial-temporal graph is its capacity to simultaneously account for both spatial and temporal dependencies.
To build a spatial-temporal graph, we commence by formulating a sequence of spatial graphs. For every step t, a spatial graph G t is constructed to represent the locations of pedestrians within a given scene. Each G t is composed of three parts: a set of vertices V t , a set of edges E t , and an adjacency matrix A t , i.e., G t = (V t , E t , A t ).
Elaborating further, the vertex set is given by
{v n t | ∀n ∈ {1, . . . , N}}, where each vertex v n t represents a pedestrian. The edge set E t is composed of edges between two vertices, i.e., E t = {e mn t | ∀m, n ∈ {1, . . . , N}}. Specifically, the edge e mn t represents whether the vertex v m t and the vertex v n t are connected or not. If v m t and v n t are connected, e mn t = 1; if v m t and v n t are not connected, e mn t = 0. The adjacency matrix A t ∈ R N×N is defined as follows: A t = a 11 t a 12 t . . . a 1N t a 21 t a 22 t . . . a 2N t . . . . . . . . . . . . a N1 t a N2 t . . . a NN t
Each item a mn t (m, n ∈ {1, . . . , N}) models the influence between the vertex v m t and the vertex v n t . As such, a spatial-temporal graph G 1:T is finally constructed, consisting of a series of spatial graphs G 1:T = {G 1 , . . . , G T }.
Based on the spatial-temporal graph representation developed in Section 3.1.1, we introduce the spatial-temporal graph convolution (ST-GC) operation in this subsection. Before diving into the complete technical details of ST-GC, we start with a more general version of the convolution operation, which is on a two-dimensional grid or feature map.
In deep learning, neural networks that employ convolution operations are referred to as "convolutional neural networks (CNNs)" [49,53]. Typically, a CNN consists of multiple convolutional layers, and within each layer, multiple learnable filters or kernels are applied to the input feature map. These filters help to capture local patterns or features, enabling CNNs to learn hierarchical representations from the input data. Additionally, CNNs significantly benefit from the idea of "parameter sharing", which reduces the number of parameters compared to fully connected layers.
For example, suppose the kernel size is equal to k, feature (l) denotes the feature map at layer l, and feature (l+1) denotes the feature map at layer l + 1. Through the training process, convolution operations are able to learn to aggregate the information from the neighbors centering around each location in feature (l) . Therefore, the convolution operation on layer l can be summarized as:
Typically, an attention mechanism consists of three major components: a query, a key, and a value [72]. Intuitively speaking, the query is a representation of the current item or context that the model is trying to process. It serves as a reference for determining essential elements within the input data. The key represents an item in the input sequence, which the model compares with the query to determine their similarity or relevance. The value represents the significant information associated with each element in the input data.
The attention mechanism computes a score for each key-query pair, typically using a dot product, scaled exponential, or some other similarity function [71][72][73]79,80]. These scores are then passed through a softmax function and are converted into a probability distribution, which represents "the attention weights". Intuitively speaking, when a weight between a query and a key is higher, the corresponding key-value pair is more important; thus, the attention mechanism pays more "attention" to this pair.
Finally, the attention mechanism computes a weighted sum of the values using the obtained attention weights, effectively selecting and aggregating the relevant information from different elements. This aggregated context vector is then used in subsequent layers of the model to make predictions. The process can be summarized into the following equation:
where d k is the dimension of each query, and the term 1/ √ d k can enhance the numerical stability of attention mechanisms.
In the context of GNNs, the previously mentioned three components, namely, a query, a key, and a value, are employed to learn meaningful representations of nodes, based on their local features and the structure of the underlying graph. As discussed in Section 3.1.1, each vertex represents a pedestrian, and v n t represents pedestrian n at step t. We represent his or her corresponding query vector as qry n t = f Qry (v n t ), the key vector as key n t = f Key (v n t ), and the value vector as val n t = f Val (v n t ). As illustrated in Figure 4, there are two major kinds of attention mechanisms in our framework: spatial attention and temporal attention. Both of them can be viewed as a way of message passing on a connected graph [73,81].
Spatial attention focuses more on message exchanges among nodes within the same time step. Intuitively speaking, it helps to generate feasible trajectories by aggregating information from other pedestrians. Suppose the message passed from node v m t to v n t is msg m→n t , which is defined as:
and based on Equation (8), we may formulate spatial attention as:
On the other hand, temporal attention focuses more on the process of temporal message passing by aggregating information through the relevant time steps. Intuitively speaking, it helps to generate feasible trajectories by incorporating the temporal significant features of pedestrians.
Here, we show how the temporal message of v n t passed from t ′ to t:
and based on Equation ( 8), we may formulate temporal attention as follows: stands for the message passed from node v m t to v n t , and msg n t ′ →t stands for the temporal message of v n t passed from t ′ to t. There are two primary types of attention mechanisms in our framework: spatial attention and temporal attention. (a) Spatial attention models the crowd as a graph and helps to predict a pedestrian's trajectory based on the movements of her or his neighboring pedestrians. (b) Temporal attention, on the other hand, focuses on each individual pedestrian and primarily assists in capturing her or his trajectory trends over time.
Difference Between ST-GC and ST-GAT Spatial-temporal graph convolution (ST-GC) and spatial-temporal graph attention (ST-GAT) are two commonly used techniques in GNNs. Both of them can learn meaningful representations of pedestrian trajectories through the spatial and temporal dimensions. Their major difference lies in how they aggregate information from neighboring nodes.
In ST-GC, information from neighboring nodes is communicated by applying convolution filters or kernels on the graph, which typically involves a weighted sum of features across neighboring nodes. Usually, those weights can be identical (e.g., GraphSAGE [63]), predetermined, or learnable ( [60,70]). Therefore, the weights are considered to be "explicitly" assigned to the neighborhoods of the focused node during the aggregation process [59]. • However, in ST-GAT, the weights between two connected nodes are considered to be "implicitly" computed. Specifically, those weights are learned based on the similarity of their feature representations, which takes into account the relative importance for different node pairs [59,73]. Typically, more important nodes tend to have higher similarity scores, resulting in them being assigned larger weights.
In this subsection, based on the discussed ST-GC and ST-GAT, we aim to clarify how they capture the modeling of pedestrian social interactions.
As introduced in Section 3.1.1, the adjacency matrix A t can be considered as a representation of the graph edge attributes. In the spatial-temporal graph convolution part, we incorporate our prior knowledge about the social relations among different pedestrians into a kernel function, e.g., pedestrians closer in distance tend to be more important. The kernel function maps node attributes at v n t and v m t to the attribute value a mn t , which is defined as follows:
Equation ( 13) is consistent with the intuition that pedestrians are more likely to be influenced by each other if they are closer. Additionally, the kernel function is symmetric:
However, since the form of the kernel function is predetermined, and some pedestrian interactions are asymmetric, the interaction modeling in a purely spatial-temporal graph convolution-based model might be insufficient. Therefore, we integrate the spatial-temporal graph attention mechanism into our model. As discussed in Section 3.1.3, the relative importance of pedestrian m to pedestrian n is data-dependent, which is determined by the value of msg m→n t = qry n t • key m t . Therefore, there is not any predetermined kernel in the the spatial-temporal graph attention mechanism. Generally, the spatial-temporal graph attention mechanism is able to capture the asymmetric importance:
which stems from the fact that, in general, qry m t ̸ = qry n t and key m t ̸ = key n t .
In real-world scenarios, when a pedestrian encounters other pedestrians, their reactions can vary from person to person. This variance can be influenced by factors like age, with different age groups having distinct walking behaviors [24]. Furthermore, an individual's walking preference might have notable changes depending on whether walking alone or in a group [22,25]. Although few frameworks have been employed to produce such diverse trajectory styles (e.g., the variety loss [34]), there is still a need to understand how to effectively capture the intrinsic heterogeneity within the pedestrian trajectories.
In this subsection, to solve the mentioned issue, we introduce the technique of variational mutual information maximization. We begin by considering the principles of information theory [82][83][84][85]. Suppose X and Y are random variables. If we want to measure the "amount of information" learned about Y by providing the knowledge of X or vice versa, mutual information I(X; Y) is utilized. The mutual information I(X; Y) can be expressed as the difference between the self-entropy of Y and the conditional entropy of Y given X:
where H(X) denotes the self-entropy of X, and H(Y) denotes the self-entropy of Y. H(X | Y) denotes the conditional entropy of X given Y, and H(Y | X) denotes the conditional entropy of Y given X.
The conditional mutual information is defined as below:
Intuitively speaking, I(X; Y | Z) can be treated as how much uncertainty is reduced in X when Y is observed, given Z. If X and Y are independent, the knowledge of X does not provide any information about Y and vice versa. As a result, the mutual information between X and Y is zero. However, given Z, if the knowledge of X provides extensive information about Y, then I(X; Y | Z) can be extremely high.
This interpretation helps to formulate the idea: given the past trajectory tr n obs , in order to learn meaningful representations for the future pedestrian trajectory, there should be high conditional mutual information between the latent code c n and the generated trajectory G(tr n obs , c n ). In other words, I(c n ; G(tr n obs , c n ) | tr n obs ) should be high. As such, we propose an information-theoretic loss:
Based on Equations ( 19) and (20), we are able to derive the following equation:
c|tr n obs ,tr n pred ) log P c ′ | tr n obs , tr n pred + H(c n ) =E tr n pred ∼G D KL (P(• | tr n obs , tr n pred ) ∥ Q φ (• | tr n obs , tr n pred )) + E c ′ ∼P(c|tr n obs ,tr n pred ) log Q φ (c ′ | tr n obs , tr n pred ) + H(c n ) ≥E tr n pred ∼G E c ′ ∼P(c|tr n obs ,tr n pred ) log Q φ (c ′ | tr n obs , tr n pred ) + H(c n ) (22) However, in practice, directly maximizing the mutual information I(c n ; G(tr n obs , c n ) | tr n obs ) is extremely challenging, as it requires the truth unknown posterior P(c n | tr n obs , tr n pred ). Therefore, we utilize a common technique in statistics and machine learning to address this problem, i.e., variational inference [85-88]. By defining an approximate posterior Q φ (c n | tr n obs , tr n pred ) over the original unknown posterior P(c n | tr n obs , tr n pred ), we are able to construct a lower bound over the original quantity -H(c n | G(tr n obs , c n ), tr n obs ): -H(c n | G, tr n obs ) = E tr n pred ∼G E c n ∼P(c n |tr n obs ,tr n pred ) log P c n | tr n obs , tr n pred
where G is the abbreviation for G(tr n obs , c n ), D KL (• ∥ •) stands for the Kullback-Leibler (KL) divergence, and the last step holds true because KL divergence is always always non-negative [82,84]. Therefore, we may construct a lower bound L I over the original objective Equation (21):
As the approximate posterior Q φ (c n | tr n obs , tr n pred ) approaches the true posterior dis- tribution P(c n | tr n obs , tr n pred ), D KL (P(• | tr n obs , tr n pred ) ∥ Q φ (• | tr n obs , tr n pred )) approaches zero. Therefore, the lower bound L I approaches the mutual information I(c n ; G(tr n obs , c n ) | tr n obs ) and becomes tighter. It is worth mentioning that we also optimize the conditional entropy of the latent code H(c n | tr n obs ), so that we the latent variable distribution and the predictor are learned simultaneously.
To summarize, the final objective of the variational mutual information maximization part can be written as:
where P θ (c n | tr n obs ) is the conditional prior distribution for the latent code c n . The primary differences between Equation ( 20) and the mutual information-inspired objective in [85] are:
1.
In ref. [85], there is only one latent code for each training example. However, in this paper, there are multiple latent codes for each training example. Different pedestrians may have distinct preferences and walking styles. It is generally infeasible to assume all pedestrians follow the same preference or walking style. Therefore, for each pedestrian n, he or she has its own latent code c n , and different pedestrians generally have different latent codes, allowing the proposed framework to effectively model the latent patterns in pedestrian trajectories.
In this paper, the proposed information-theoretic loss is based on the conditional mutual information. However, in ref. [85], the loss is based on the mutual information.
Different from the previous research taken in [85], where the prior latent code distribution is assumed to be fixed, we opt to optimize the prior distribution P θ (c n | tr n obs ) as well.
To optimize the proposed method, we utilize the multi-objective loss defined below:
where L pred denotes the prediction loss, L GAN denotes the generative adversarial loss, and L info denotes the information loss. λ 1 , λ 2 and λ 3 control the relative importance of each loss.
• L pred : The prediction loss relies on negative log-likelihood, which is defined as:
where c n ∼ P θ (c n | tr n obs ). Intuitively speaking, when L pred is decreasing, the log- likelihood of tr n pred is increasing. The model G(tr n obs , c n ) and the posterior distribution P θ (c n | tr n obs ) together are encouraged to accurately predict the ground-truth future trajectory.
• L GAN : The generative adversarial loss relies on the generator G and the discriminator D, in which two models are jointly trained. The generator G captures the distribution for the future trajectory, and the discriminator distinguishes whether a sample comes from the training data or the generator G.
• L info : The information-theoretic loss relies on the conditional prior distribution P θ (c n | tr n obs ), the model G(tr n obs , c n ), and the approximate posterior Q φ (c n | tr n obs , tr n pred ), which has been discussed in Section 3.2.
UNIV. These datasets consist of real-world pedestrian trajectories with complex human interactions. Specifically, lots of challenging pedestrian behaviors are covered in the datasets, such as pedestrians crossing each other, walking together, avoiding collision, and groups assembling and disbanding [91]. In accordance with a similar strategy utilized in previous studies [27,34], all trajectories are sampled every 0.4 s.
In accordance with prior work [27,34,57,93], we choose to use the following evaluation metrics:
Average Displacement Error (ADE): The average L 2 distance between the predicted trajectory and the ground truth trajectory across all time steps, which is defined as follows:
where (x n t , y n t ) are the real locations, and ( xn t , ŷn t ) are the predicted locations. • Final Displacement Error (FDE): The L 2 distance between the predicted final destination and the true final destination at the end of the prediction period T pred , which is defined as follows:
Intuitively speaking, different metrics serve as different purposes. ADE evaluates the average prediction error across the whole trajectory, whereas FDE focuses solely on the prediction error at the destination.
In this section, we provide important details on appropriately implementing the proposed model. To facilitate the learning process [60,70], we normalize the adjacency matrix A t at each time step t as follows:
where I is an identity matrix, which serves to add self-connections to all nodes. Λ t is the diagonal node degree matrix of (A t + I). We use A to denote the stack of all adjacency matrices A 1 + I, . . . , A T + I, and Λ to denote the stack of matrices Λ 1 , . . . , Λ T . Suppose the vertices values at the layer l as V (l) , which is a stack of vertices values across all steps 1, . . . , T. We can now employ the matrices defined to implement the ST-GC layers:
where W (l) represents the learnable parameters at the l-th layer. The above Equation (38) follows similar ideas in [57,70].
The proposed model consists of a series of ST-GC and ST-GAT layers, which helps to extract spatial-temporal node embeddings from the input data. Later, those node embeddings are concatenated with latent codes, and then, several convolutional layers are followed such that the output time dimension is manipulated to match the length of predicted horizon T pred .
Unless noted otherwise, we choose to use PReLU [94] as the activation function through our model. During training, we used a batch size of 128 and the default optimizers were chosen to use Stochastic Gradient Descent (SGD). The initial learning rate was 0.01, and it was decreased based on exponential scheduling with a decay factor 0.97. To prevent overfitting to the training data, we randomly dropped out the features at a probability of 0.5.
In this subsection, we begin by comparing our results with baseline models. Subsequently, we provide a comprehensive qualitative analysis of how our proposed method models pedestrian interactions and takes heterogeneous behavior choices into account. We illustrate cases where InfoSTGCAN successfully predicts collision-free trajectories for scenarios such as pedestrians walking in the same direction, approaching from opposing directions, or merging at angles. Moreover, our model is able to generate socially acceptable trajectories based on the predicted personalized latent codes.
Baselines We compare the proposed method with the following baselines:
1.
Linear: A linear regression model characterized by minimizing the least square error.
Social LSTM (S-LSTM) [27]: An LSTM approach that incorporates the "social pooling" mechanism for hidden states.
S-GAN-Pooling [34]: A GAN-based approach that utilizes global pooling for pedestrian interactions. 4.
SR-LSTM-2 [29]: An LSTM-based method that leverages a state refinement technique. 5.
GAT [55]: A graph attention network leveraging the sequence-to-sequence architecture. 6.
Sophie [35]: A GAN-based method that takes both scene and social factors into account through a dual attention mechanism. 7.
SCAN [58]: An LSTM-based encoder-decoder framework that incorporates a novel spatial attention mechanism to predict trajectories for all pedestrians. 8.
Social-STGCNN [57]: A spatial-temporal graph-based approach that employs a spatialtemporal graph convolutional network to handle complex social interactions.
The performance of the proposed method is evaluated against other benchmark models on ADE/FDE metrics, as presented in Table 2. In general, our method outperforms all baseline methods on the two metrics. Our proposed model achieves an error of 0.62 on the average FDE metric, representing an approximate 20% improvement over the previous best performance (0.75). For the average ADE metric, the proposed model is better than the previous best performance by 5%. Interestingly, although our model does not need the vision signal containing scene context information, it can still outperform methods that utilize such information, such as SR-LSTM and Sophie. Table 3. The ablation study on λ. Multiple different values are tested to show the model performance on the HOTEL dataset. These findings validate the importance of the GAN loss component and the significance of maintaining a balanced weight between the prediction loss L pred and the information loss L info for optimal performance.
In this paper, we formulate the task of pedestrian trajectory prediction as a spatialtemporal graph and develop a novel pedestrian trajectory prediction model, InfoSTGCAN. The proposed model takes into account both pedestrian interactions and heterogeneous behavior choices. Specifically, to better model pedestrian interactions, our proposed model consists of two parts, spatial-temporal graph convolution and spatial-temporal graph attention, enabling the analysis of interactions through a combination of prior knowledge and data-driven methods. To address the heterogeneity within the pedestrian behavior choices, we utilize the variational mutual information maximization technique, which is primarily composed of a conditional prior distribution and an approximate posterior distribution.
The proposed method outperforms baseline models across several publicly accessible datasets. Visualization of the generated trajectories reveals our method's capacity to handle various scenarios, including pedestrians going straight from different directions or making a right turn first and then going straight. We also conduct a qualitative analysis of the proposed method in different situations, such as collision avoidance, parallel walking, and pedestrians merging. In these situations, InfoSTGCAN tends to generate realistic collision-free trajectories. Additionally, we show that our framework is able to generate satisfactory trajectories through learning a personalized pedestrian-level latent code.
Nevertheless, we identify several promising future directions that are worth exploring further. The first aspect involves exploring more metrics related to probabilistic trajectory prediction beyond the standard ADE/FDE for training and evaluation, e.g., Mahalanobis distance [95]. Secondly, our methodology currently models pedestrian social interactions through ST-GC and ST-GAT; an exciting direction is to integrate more socially aware or physics-based methods [96]. Lastly, the third aspect refers to an integrative approach that combines heuristic optimization [97], causal inference [22,23,26] or clustering techniques [98].