Speaker tracking is the process of assigning an unknown speech utterance to one of known speakers in a set of target speakers, including following the speaker's voice over time in the course of an audio stream. The first step in speaker tracking involves classifying speakers and identifying the target speaker. The goal is to identify all speech segments uttered by the same speaker in an audio recording and assign unique labels. The second step involves tracking speaker of interest throughout the audio segment. For our work, we also identify roles of our speakers using their speech duration. To perform speaker tracking, good representations of data and features that reflect the semantic meaning are required to produce a robust model.
In our experiments, we employ CRSS-UTDallas Fearless Steps Apollo 11 audio corpus 1 consisting of +9k audio data (100 hr hand labelled) involving more than +400 personnel serving as mission specialists who communicate across 30 audio loops [1,2]. Each channel reflects a single communications loop (channel) that can contain anywhere from 3-65 speakers over extended time periods. Due to strict NASA communication protocols in such time-critical missions, most personnel employed a compact speaking style, with information turn-taking over 3-5sec windows [3]. Furthermore, some speakers had less than 3 seconds of utterance duration. This poses a unique and challenging research problem of finding 'needles in a haystack' from a speaker tracking perspective [1,4,5].
Performing speaker classification on short duration utterances is challenging as such utterances do not contain enough contextual information to accurately classify the speaker. Previous work [6,7] have explored building robust models for short duration utterance, by extracting speaker specific features. However, this requires a substantial amount of training data to perform well and they do not consider varying duration utterances. [8] proposes an approach that works on varying duration speech data by aggregating information across multiple utterances, although this system can handle varying duration speech data, it may not work well with short duration utteranced.
Graph Neural Networks (GNNs) have rapidly developed with powerful variants such as Graph Convolutional Network (GCN) [9], Graph Attention Network (GAT) [10], and Graph-SAGE [11]. Despite their success, GNNs have not been used or studied often in the context of speaker classification or speaker tracking. [12] proposes a graph convolution network for speaker verification and uses attention mechanism to obtain speaker representations. However, this model requires large amounts of training data and does not consider short duration utterance. Hence, in our study, we propose using a variation of the Graph Attention Network (GAT) framework with a dynamic attention, which can handle varying duration utterance and works with a small training dataset, while remaining robust to noise. ers that are selected based on the presence of atleast 3 speaker utterances;[iii] Using the concept of 'Finding Waldo' to identify and track key speakers of interest (SOI) :Flight Director (FD), Capsule Communicator (CAPCOM), Guidance, Navigation and, Control (GNC), Electrical, Environmental, and Consumables Manager (EECOM), and Network (NTWK) on three mission critical phases and compare their speech duration on different phases;[iv] Contributing to archiving and serving as a lasting tribute to the 'Heroes Behind the Heroes of Apollo', thus preserving the "words spoken in space".
The UTDallas Fearless-Steps Apollo corpus comprises of 19,000 hours of audio, which presents unique and multiple challenges due to severe noise and degradation, as well as overlapping instances over 30 channels. For our work, we selected a subset of 100 hours [13,14,15], which were manually transcribed by professional annotators for speaker labels. The 100 hrs were obtained from three mission-critical events: Lift-Off (25 hours), Lunar-Landing (50 hours), and Lunar-Walking (25 hours).
Out of 30 channels, we selected five channels with the most speech activity over the selected events: Flight Director (FD), Mission Operations Control Room (MOCR), Guidance Navigation and Control (GNC), Network Controller (NTWK), and Electrical, Environmental, and Consumables Manager (EECOM).
Although the corpus contains 100 hours of audio data, the total amount of actual speech content is approximately 17 hours.
This system is based on a Gausian Mixture Model-Universal background Model (GMM-UBM) system [17,18], which serves as the acoustic-feature system. Here, the UBM model is trained on the NIST SRE 16 corpus to create a 2048 component full-covariance GMM. A 600 dim i-Vector speaker embedding is developed and extracted.
To extract x-Vectors, a feed-forward Deep Neural Network (DNN) computes the speaker embeddings from variable-length acoustic segments [19,20]. The DNN embeddings are trained on the SRE16 dataset and extracted x-Vectors are 512 dim vectors. The Kaldi speech recognition toolkit [21] was used to train both i-Vectors and x-Vectors.
Emphasized Channel Attention, Propagation and Aggregation (ECAPA) Time Delay Neural Networks is a deep learning architecture which combines time-delay neural networks (TDNNs) and convolutional neural networks (CNNs). The speaker embeddings are extracted from the output of the bottleneck layer resulting in a 192 dim vector. The system is pretrained on Vox-celeb1+Voxceleb2 training data. The embeddings are extracted using attentive statistical pooling [22,23].
A graph is defined by its node set V = {v1, ....vn} and edge set E ⊆ {(vi, vj) | vi, vj ∈ V } where (i, j) ∈ E denotes an edge from node j to node i. Each node has a node feature vector and associated with other nodes by it's edges. A message exchange is performed at each round where a node sends messages to its neighbors, and aggregates incoming messages from its neighbors through a message function f(.). Each graph has a unique message passing function and aggregation function AGG(.) [24].
For our experiments, we consider popular GNN variants such as the Graph Convolutional Network (GCN) [9], Graph Attention Network (GAT) [10], Graph Neural Network with convolutional auto-regressive moving average filters (ARMA) [25], and GraphSAGE [11]. These four GNNs are all considered.
Graph Attention Networks (GAT) use attention mechanisms to model the interaction between nodes in a graph. To learn the node representation for each node, attention coefficients between pairs of nodes are computed to weigh the contribution of each neighbor representation of the target node. The output of the final layer are a set of new features for each node [10]. The propagation function can be defined as:
where Ni is the set of neighboring nodes of node i, and h ′ i is the updated feature representation of node i. Let hi ∈ R F be the input features of node i and hj ∈ R F be the input features of a neighboring node j. The normalized attention coefficients are used to compute a linear combination of the features corresponding to them. To serve as the final output features for every 1: Speaker classification accuracy on a range of speakers using several frameworks node, the attention mechanism is therefore defined as:
where a ∈ R 2F is a trainable attention parameter vector, W ∈ R F ×F ′ is a trainable weight matrix, || denotes concatenation, and LeakyReLU is a non-linearity.
The motivation for GAT is to compute a representation for every node as a weighted average of its neighbors. However, GAT is severely constrained, because it can only calculate the static attention. This means that the attention function always weighs one key at least as much as any other key, regardless of the query. This limitation can be problematic when attempting to fit to the available training data, as the model may not be able to focus on the most relevant inputs. Furthermore, our corpus consists of varying duration utterance where it may be required to assign alternate weights to different duration utterances. To address varying duration utterances, we make use of an edge softmax function; a normalization function that converts edge weights into a probability distribution, allowing all nodes to contribute to the representation, albeit with different weights. The attention score eij on the nodes can then be computed as:
This equation indicates the importance of node j's features to node i. eij is only computed for nodes j ∈ Ni, where Ni is some neighborhood of node i in the graph. Given a set of edge weights, the softmax function normalizes the weights such that they add up to one and is given as:
where softmax(eij) is the normalized weight for the edge connecting node i and node j, and eij is the unnormalized weight for that edge. The edge softmax function can make use of contextual information from both shorter and longer duration utterances by effectively learning the underlying graph structure, and thereby capturing important patterns in the data. This helps to prevent over-emphasizing longer utterance duration, by giving appropriate weights to all nodes in the graph. To resolve the static attention problem, our framework will be using dynamic attention in GAT [26]. To create a dynamic graph attention network, the order of the internal operators in the attention coefficient function is modified and given as:
The proposed modification in this study significantly enhances the robustness of the GAT function by allowing it to decay noisy edges, which leads to better performance in the presence of edge noise. For our study, this modification is particularly useful since the nodes have varying duration as it can prioritize nodes with longer duration, while assigning lower attention scores to those with shorter duration. Using the edge softmax prevents assigning extremely low attention scores for the shorter duration nodes, in turn learning a better generalized node representation. In addition, we propose using normalization techniques such as batch normalization and layer normalization. Batch normalization [27] improves the stability of the attention weights by normalizing the nodes to each layer of the network, allowing the network to handle channel noise in the input data. Layer normalization [28] can also be used to normalize the outputs of the attention layer to help regularize the network and prevent overfitting. Overall, the above techniques are expected to help in improving the effectiveness, robustness, and stability of the model.
For this study, a graph is built for every 30 minute audio segment, where each node represents a speaker utterance which can vary from 1s to 15s, meaning that every graph can have a variable number of nodes. The features of each node vi is represented by extracted speaker embeddings denoted by F dimensions depending on the type of speaker embedding xi, i ∈ 1, 2, ...., N where N represents the number of nodes/speaker utterances. For our proposed framework, we construct a 2 layer GATv2 network using PyTorch Geometric [29]. After every GATv2 layer, we use a batch normalization and layer normalization. Each GATv2 layer has an edge softmax function. Finally, we use a cross entropy loss to perform classification on the speakers. Each network mentioned in Sec 4, including our proposed framework, is a 2 layer network that uses a cross entropy loss trained with an Adam optimizer with a learning rate of 0.0001 with a batch size of 13. Note that every graph will have a different number of nodes depending on the number of speaker utterances in that 30 minute audio segment.
The performance of our proposed framework was evaluated with three different pretrained speaker embeddings: i-Vector, x-Vector, and ECAPA-TDNN. We compare our network with the baseline systems and other popular Graph Networks. Additionally, we test the networks against different number of speakers to determine its efficacy on small vs large amounts of data. Starting with 10 speakers who had the highest amount of speech data, we progressively the number of speakers to 140 speakers. For each case, our network framework outperforms the baseline models and the popular Graph models ranging from 90.74% speaker classification accuracy for 10 speakers to 79.86% speaker classification accuracy for 140 speakers. As the number of speakers increase, the classification accuracy drops. As seen in Fig 1, this corpus duration labels are highly imbalanced among speakers, with speech duration per speaker decreasing as we increase the number of speakers. This explains the drop in classification accuracy as we increase the number of speakers, since the training model does not have sufficient number of examples to accurately classify every test sample.
As noted, there are three mission critical phases: Lift Off(52 speakers, 25hrs), Lunar Landing(92 speakers, 50 hrs), and Lunar Walking(37 speakers, 25 hrs). The Green team was responsible for Lift Off phase and the Lunar Walking phase. Hence, we see the same speakers operating these phases in Fig 3 . The donut plots are a new visualization tool to analyze primary speaker duration (speaker in the center, highlighted by green) vs other speakers (speakers that interact with primary speakers, highlighted by orange). For all three phases, we observe that Flight Director has a longer duration (73%, 67%, and 29% of primary speaker duration for Lift Off, Lunar Landing, and Lunar Walking) compared to his secondary speakers (includes multiple speakers) and other S-o-I. The CAPCOM was the only flight controller authorized to communicate directly with the spacecraft's crew. Therefore, the secondary speakers (i.e., astronauts) are more active in this channel than the CAPCOM (52%, 40%, and 23% of primary speaker duration for Lift Off, Lunar Landing, and Lunar Walking). With our speaker tracking, we assigned names to each speaker instead of using their speaker handle since each speaker handle represents 3 to 4 individuals, who worked in a day. This approach allows us to honor the "Heroes Behind the Heroes", and archive the speech of these individuals for posterity.
In this study, we proposed a solution to the problem of varying utterance duration in the Fearless Steps Apollo audio corpus by using a Graph Attention Network framework with dynamic attention. Furthermore, we assessed the performance of our proposed network on a range of speakers, from small to large and highly imbalanced duration and speaker count dataset, and demonstrate that it outperforms all baseline models and other popular Graph Networks. Additionally, we use the concept of 'Finding Waldo' to track and tag speaker-of-interest during three critical phases of the mission, thus providing recognition to individuals who contributed to the success of the mission. We analyze the speaker duration of primary and secondary speakers using donut plots, which reveal an intriguing global perspective of speaker interactions between NASA mission specialists. Ultimately, our analysis of key speakers-of-interest serves as a lasting tribute to the "Heroes Behind the Heroes of Apollo".