More Like this

1. MONET: Multiview Semi-supervised Keypoint via Epipolar Divergence

   https://doi.org/10.1109/ICCV.2019.00084  

   Yao, Y.; Jafarian, Y.; Park, H.S. (January 2019, International conference on Computer Vision)

   This paper presents MONET -- an end-to-end semi-supervised learning framework for a keypoint detector using multiview image streams. In particular, we consider general subjects such as non-human species where attaining a large scale annotated dataset is challenging. While multiview geometry can be used to self-supervise the unlabeled data, integrating the geometry into learning a keypoint detector is challenging due to representation mismatch. We address this mismatch by formulating a new differentiable representation of the epipolar constraint called epipolar divergence---a generalized distance from the epipolar lines to the corresponding keypoint distribution. Epipolar divergence characterizes when two view keypoint distributions produce zero reprojection error. We design a twin network that minimizes the epipolar divergence through stereo rectification that can significantly alleviate computational complexity and sampling aliasing in training. We demonstrate that our framework can localize customized keypoints of diverse species, e.g., humans, dogs, and monkeys. 

   more » « less
2. MONET: Multiview Semi-supervised Keypoint Detection via Epipolar Divergence

   https://doi.org/10.1109/ICCV.2019.00084  

   Yao, Y.; Jafarian, Y.; Park, H.S. (January 2019, International conference on computer vision)

   This paper presents MONET -- an end-to-end semi-supervised learning framework for a keypoint detector using multiview image streams. In particular, we consider general subjects such as non-human species where attaining a large scale annotated dataset is challenging. While multiview geometry can be used to self-supervise the unlabeled data, integrating the geometry into learning a keypoint detector is challenging due to representation mismatch. We address this mismatch by formulating a new differentiable representation of the epipolar constraint called epipolar divergence---a generalized distance from the epipolar lines to the corresponding keypoint distribution. Epipolar divergence characterizes when two view keypoint distributions produce zero reprojection error. We design a twin network that minimizes the epipolar divergence through stereo rectification that can significantly alleviate computational complexity and sampling aliasing in training. We demonstrate that our framework can localize customized keypoints of diverse species, e.g., humans, dogs, and monkeys. 

   more » « less
3. Uncertainty-aware Self-supervised 3D Data Association

   Wang, Jianren; Ancha, Siddharth; Chen, Yi-Ting; Held, David (January 2020, Proceedings of the IEEERSJ International Conference on Intelligent Robots and Systems)

   null (Ed.)

   3D object trackers usually require training on large amounts of annotated data that is expensive and time-consuming to collect. Instead, we propose leveraging vast unlabeled datasets by self-supervised metric learning of 3D object trackers, with a focus on data association. Large scale annotations for unlabeled data are cheaply obtained by automatic object detection and association across frames. We show how these self-supervised annotations can be used in a principled manner to learn point-cloud embeddings that are effective for 3D tracking. We estimate and incorporate uncertainty in self-supervised tracking to learn more robust embeddings, without needing any labeled data. We design embeddings to differentiate objects across frames, and learn them using uncertainty-aware self-supervised training. Finally, we demonstrate their ability to perform accurate data association across frames, towards effective and accurate 3D tracking. 

   more » « less
4. Semi-Supervised Learning for Wearable-based Momentary Stress Detection in the Wild

   https://doi.org/10.1145/3596246  

   Yu, Han; Sano, Akane (June 2023, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies)

   Physiological and behavioral data collected from wearable or mobile sensors have been used to estimate self-reported stress levels. Since stress annotation usually relies on self-reports during the study, a limited amount of labeled data can be an obstacle to developing accurate and generalized stress-predicting models. On the other hand, the sensors can continuously capture signals without annotations. This work investigates leveraging unlabeled wearable sensor data for stress detection in the wild. We propose a two-stage semi-supervised learning framework that leverages wearable sensor data to help with stress detection. The proposed structure consists of an auto-encoder pre-training method for learning information from unlabeled data and the consistency regularization approach to enhance the robustness of the model. Besides, we propose a novel active sampling method for selecting unlabeled samples to avoid introducing redundant information to the model. We validate these methods using two datasets with physiological signals and stress labels collected in the wild, as well as four human activity recognition (HAR) datasets to evaluate the generality of the proposed method. Our approach demonstrated competitive results for stress detection, improving stress classification performance by approximately 7% to 10% on the stress detection datasets compared to the baseline supervised learning models. Furthermore, the ablation study we conducted for the HAR tasks supported the effectiveness of our methods. Our approach showed comparable performance to state-of-the-art semi-supervised learning methods for both stress detection and HAR tasks. 

   more » « less
5. Multiview Aerial Visual Recognition (MAVREC): Can Multi-View Improve Aerial Visual Perception?

   https://doi.org/10.1109/CVPR52733.2024.02140  

   Dutta, Aritra; Das, Srijan; Nielsen, Jacob; Chakraborty, Rajatsubhra; Shah, Mubarak (June 2024, IEEE)

   Despite the commercial abundance of UAVs, aerial data acquisition remains challenging, and the existing Asia and North America-centric open-source UAV datasets are small- scale or low-resolution and lack diversity in scene contextuality. Additionally, the color content of the scenes, solar zenith angle, and population density of different geographies influence the data diversity. These factors conjointly render suboptimal aerial-visual perception of the deep neural net- work (DNN) models trained primarily on the ground view data, including the open-world foundational models. To pave the way for a transformative era of aerial detection, we present Multiview Aerial Visual RECognition or MAVREC, a video dataset where we record synchronized scenes from different perspectives — ground camera and drone-mounted camera. MAVREC consists of around 2.5 hours of industry-standard 2.7K resolution video sequences, more than 0.5 million frames, and 1.1 million annotated bounding boxes. This makes MAVREC the largest ground and aerial view dataset, and the fourth largest among all drone-based datasets across all modalities and tasks. Through our extensive benchmarking on MAVREC, we recognize that augmenting object detectors with ground view images from the corresponding geographical location is a superior pre-training strategy for aerial detection. Building on this strategy, we benchmark MAVREC with a curriculum- based semi-supervised object detection approach that lever- ages labeled (ground and aerial) and unlabeled (only aerial) images to enhance aerial detection. 

   more » « less