More Like this

1. 3D Reconstruction of Objects in Hands Without Real World 3D Supervision

   https://doi.org/10.1007/978-3-031-73229-4_8  

   Prakash, Aditya; Chang, Matthew; Jin, Matthew; Tu, Ruisen; Gupta, Saurabh (October 2024, Springer Nature Switzerland)

   Prior works for reconstructing hand-held objects from a single image train models on images paired with 3D shapes. Such data is challenging to gather in the real world at scale. Consequently, these approaches do not generalize well when presented with novel objects in in-the-wild settings. While 3D supervision is a major bottleneck, there is an abundance of a) in-the-wild raw video data showing hand-object interactions and b) synthetic 3D shape collections. In this paper, we propose modules to leverage 3D supervision from these sources to scale up the learning of models for reconstructing hand-held objects. Specifically, we extract multiview 2D mask supervision from videos and 3D shape priors from shape collections. We use these indirect 3D cues to train occupancy networks that predict the 3D shape of objects from a single RGB image. Our experiments in the challenging object generalization setting on in-the-wild MOW dataset show 11.6% relative improvement over models trained with 3D supervision on existing datasets. 

   more » « less
2. Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing

   https://doi.org/10.1109/CVPR.2017.49  

   Li, Chi; Zia, M. Zeeshan; Tran, Quoc-Huy; Yu, Xiang; Hager, Gregory D.; Chandraker, Manmohan (July 2017, CVPR)

   Monocular 3D object parsing is highly desirable in various scenarios including occlusion reasoning and holistic scene interpretation. We present a deep convolutional neural network (CNN) architecture to localize semantic parts in 2D image and 3D space while inferring their visibility states, given a single RGB image. Our key insight is to exploit domain knowledge to regularize the network by deeply supervising its hidden layers, in order to sequentially infer intermediate concepts associated with the final task. To acquire training data in desired quantities with ground truth 3D shape and relevant concepts, we render 3D object CAD models to generate large-scale synthetic data and simulate challenging occlusion configurations between objects. We train the network only on synthetic data and demonstrate state-of-the-art performances on real image benchmarks including an extended version of KITTI, PASCAL VOC, PASCAL3D+ and IKEA for 2D and 3D keypoint localization and instance segmentation. The empirical results substantiate the utility of our deep supervision scheme by demonstrating effective transfer of knowledge from synthetic data to real images, resulting in less overfitting compared to standard end-to-end training. 

   more » « less
3. Towards 3D Vision with Low-Cost Single-Photon Cameras

   Mu, Fangzhou; Sifferman, Carter; Jungerman, Sacha; Li, Yiquan; Han, Mark; Gleicher, Michael; Gupta, Mohit; Li, Yin (June 2024, IEEE CVPR 2024 (Conference on Computer Vision and Pattern Recognition))

   We present a method for reconstructing 3D shape of arbitrary Lambertian objects based on measurements by miniature, energy-efficient, low-cost single-photon cameras. These cameras, operating as time resolved image sensors, illuminate the scene with a very fast pulse of diffuse light and record the shape of that pulse as it returns back from the scene at a high temporal resolution. We propose to model this image formation process, account for its non-idealities, and adapt neural rendering to reconstruct 3D geometry from a set of spatially distributed sensors with known poses. We show that our approach can successfully recover complex 3D shapes from simulated data. We further demonstrate 3D object reconstruction from real-world captures, utilizing measurements from a commodity proximity sensor. Our work draws a connection between image-based modeling and active range scanning and is a step towards 3D vision with single-photon cameras. 

   more » « less
4. MOVES: Manipulated Objects in Video Enable Segmentation

   https://doi.org/10.1109/CVPR52729.2023.00613  

   Higgins, Richard E.L.; Fouhey, David F. (June 2023, CVPR)

   Our method uses manipulation in video to learn to understand held-objects and hand-object contact. We train a system that takes a single RGB image and produces a pixel-embedding that can be used to answer grouping questions (do these two pixels go together) as well as hand-association questions (is this hand holding that pixel). Rather than painstakingly annotate segmentation masks, we observe people in realistic video data. We show that pairing epipolar geometry with modern optical flow produces simple and effective pseudo-labels for grouping. Given people segmentations, we can further associate pixels with hands to understand contact. Our system achieves competitive results on hand and hand-held object tasks. 

   more » « less
5. COHESIV: Contrastive Object and Hand Embeddings for Segmentation In Video

   Shan, Dandan; Higgins, Richard E.L.; Fouhey, David F. (January 2021, Advances in neural information processing systems)

   In this paper we learn to segment hands and hand-held objects from motion. Our system takes a single RGB image and hand location as input to segment the hand and hand-held object. For learning, we generate responsibility maps that show how well a hand’s motion explains other pixels’ motion in video. We use these responsibility maps as pseudo-labels to train a weakly-supervised neural network using an attention-based similarity loss and contrastive loss. Our system outperforms alternate methods, achieving good performance on the 100DOH, EPIC-KITCHENS, and HO3D datasets. 

   more » « less