The goal of speech separation is to extract multiple speech sources from a single microphone recording. Recently,
with the advancement of deep learning and availability of large datasets, speech separation has been formulated
as a supervised learning problem. These approaches aim to learn discriminative patterns of speech, speakers, and
background noise using a supervised learning algorithm, typically a deep neural network. A long-lasting problem
in supervised speech separation is finding the correct label for each separated speech signal, referred to as label
permutation ambiguity. Permutation ambiguity refers to the problem of determining the output-label assignment
between the separated sources and the available single-speaker speech labels. Finding the best output-label
assignment is required for calculation of separation error, which is later used for updating parameters of the
model. Recently, Permutation Invariant Training (PIT) has been shown to be a promising solution in handling the
label ambiguity problem. However, the overconfident choice of the output-label assignment by PIT results in a
sub-optimal trained model. In this work, we propose a probabilistic optimization framework to address the inefficiency
of PIT in finding the best output-label assignment. Our proposed method entitled trainable Softminimum
PIT is then employed on the same Long-Short Term Memory (LSTM) architecture used in Permutation
Invariant Training (PIT) speech separation method. The results of our experiments show that the proposed
method outperforms conventional PIT speech separation significantly (p-value < 0.01) by +1dB in Signal to
Distortion Ratio (SDR) and +1.5dB in Signal to Interference Ratio (SIR).
more »
« less
Robust Long-Term Object Tracking via Improved Discriminative Model Prediction
We propose an improved discriminative model prediction
method for robust long-term tracking based on a pre-trained short-term
tracker. The baseline pre-trained short-term tracker is SuperDiMP which
combines the bounding-box regressor of PrDiMP with the standard DiMP
classifier. Our tracker RLT-DiMP improves SuperDiMP in the follow-
ing three aspects: (1) Uncertainty reduction using random erasing: To
make our model robust, we exploit an agreement from multiple im-
ages after erasing random small rectangular areas as a certainty. And
then, we correct the tracking state of our model accordingly. (2) Ran-
dom search with spatio-temporal constraints: we propose a robust ran-
dom search method with a score penalty applied to prevent the prob-
lem of sudden detection at a distance. (3) Background augmentation for
more discriminative feature learning: We augment various backgrounds
that are not included in the search area to train a more robust model
in the background clutter. In experiments on the VOT-LT2020 bench-
mark dataset, the proposed method achieves comparable performance to
the state-of-the-art long-term trackers. The source code is available at:
https://github.com/bismex/RLT-DIMP.
more »
« less
- Award ID(s):
- 1650994
- NSF-PAR ID:
- 10289121
- Editor(s):
- Bartoli, A; Fusiello, A
- Date Published:
- Journal Name:
- Computer Vision – ECCV 2020 Workshops
- Volume:
- 12539
- Page Range / eLocation ID:
- 602-617
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Avidan, S. (Ed.)Despite the success of fully-supervised human skeleton sequence modeling, utilizing self-supervised pre-training for skeleton sequence representation learning has been an active field because acquiring task-specific skeleton annotations at large scales is difficult. Recent studies focus on learning video-level temporal and discriminative information using contrastive learning, but overlook the hierarchical spatial-temporal nature of human skeletons. Different from such superficial supervision at the video level, we propose a self-supervised hierarchical pre-training scheme incorporated into a hierarchical Transformer-based skeleton sequence encoder (Hi-TRS), to explicitly capture spatial, short-term, and long-term temporal dependencies at frame, clip, and video levels, respectively. To evaluate the proposed self-supervised pre-training scheme with Hi-TRS, we conduct extensive experiments covering three skeleton-based downstream tasks including action recognition, action detection, and motion prediction. Under both supervised and semi-supervised evaluation protocols, our method achieves the state-of-the-art performance. Additionally, we demonstrate that the prior knowledge learned by our model in the pre-training stage has strong transfer capability for different downstream tasks.more » « less
-
Self-tracking using commodity wearables such as smartwatches can help older adults reduce sedentary behaviors and engage in physical activity. However, activity recognition applications that are typically deployed in these wearables tend to be trained on datasets that best represent younger adults. We explore how our activity recognition model, a hybrid of long short-term memory and convolutional layers, pre-trained on smartwatch data from younger adults, performs on older adult data. We report results on week-long data from two older adults collected in a preliminary study in the wild with ground-truth annotations based on activPAL, a thigh-worn sensor. We find that activity recognition for older adults remains challenging even when comparing our model’s performance to state of the art deployed models such as the Google Activity Recognition API. More so, we show that models trained on younger adults tend to perform worse on older adults.more » « less
-
Language-guided smart systems can help to design next-generation human-machine interactive applications. The dense text description is one of the research areas where systems learn the semantic knowledge and visual features of each video frame and map them to describe the video's most relevant subjects and events. In this paper, we consider untrimmed sports videos as our case study. Generating dense descriptions in the sports domain to supplement journalistic works without relying on commentators and experts requires more investigation. Motivated by this, we propose an end-to-end automated text-generator, SpecTextor, that learns the semantic features from untrimmed videos of sports games and generates associated descriptive texts. The proposed approach considers the video as a sequence of frames and sequentially generates words. After splitting videos into frames, we use a pre-trained VGG-16 model for feature extraction and encoding the video frames. With these encoded frames, we posit a Long Short-Term Memory (LSTM) based attention-decoder pipeline that leverages soft-attention mechanism to map the semantic features with relevant textual descriptions to generate the explanation of the game. Because developing a comprehensive description of the game warrants training on a set of dense time-stamped captions, we leverage two available public datasets: ActivityNet Captions and Microsoft Video Description. In addition, we utilized two different decoding algorithms: beam search and greedy search and computed two evaluation metrics: BLEU and METEOR scores.more » « less
-
more » « less
Visual tracking has achieved remarkable success in recent decades, but it remains a challenging problem due to appearance variations over time and complex cluttered background. In this paper, we adopt a tracking-by-verification scheme to overcome these challenges by determining the patch in the subsequent frame that is most similar to the target template and distinctive to the background context. A multi-stream deep similarity learning network is proposed to learn the similarity comparison model. The loss function of our network encourages the distance between a positive patch in the search region and the target template to be smaller than that between positive patch and the background patches. Within the learned feature space, even if the distance between positive patches becomes large caused by the appearance change or interference of background clutter, our method can use the relative distance to distinguish the target robustly. Besides, the learned model is directly used for tracking with no need of model updating, parameter fine-tuning and can run at 45 fps on a single GPU. Our tracker achieves state-of-the-art performance on the visual tracking benchmark compared with other recent real-time-speed trackers, and shows better capability in handling background clutter, occlusion and appearance change.
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1007/978-3-030-68238-5_40
