Search for: All records

Plants are geometrically and topologically complex objects, and methods and devices that produce plant point clouds often miss parts due to self occlusions, making further analysis, such as phenotypic trait extraction or 3D reconstruction, difficult. We introduce A-Occ-Plant, a novel method for point cloud completion. The first novelty of our algorithm is converting point clouds into a set of images, which are then completed using 2D amodal segmentation. The images are then converted into a complete point cloud by using view-consistent Gaussian splats. The second novelty is the use of a coarse-to-fine hierarchical Transformer with cross-scale attention. The completed soft masks are fused into a continuous 3D density field using Gaussian splatting, removing the need for external pose estimation or fixed-size inputs. We introduce a synthetic dataset using a procedural model and a real-world plant reconstruction benchmark with artificially generated occlusions. We further benchmark A-Occ-Plant against representative 3D point-cloud completion methods, demonstrate that it recovers downstream phenotypic traits (leaf count, leaf angle, plant height), and show that it generalizes to another crops (soybean). AOcc-Plant achieves a 264.8% improvement in LPIPS and an 8.3% gain in SSIM compared to the current state of the art, while using only 2.3% of the parameters and running 39.4× faster. We release our code at https://github.com/JaeLee18/PlantPhenomics_Occlusion. 

Amodal segmentation is an image-based algorithm that aims to predict masks for both visible and occluded parts of objects. Existing methods typically rely on supervised learning with annotated amodal masks or synthetic data. The effectiveness of these methods relies heavily on the quality of the datasets. This dependence can unintentionally restrict their generalization capabilities due to insufficient diversity and size. Although existing zero-shot methods perform well on their reported datasets, their performance does not necessarily transfer to other datasets. We propose a tuning-free approach that re-purposes diffusion-based inpainting foundation models for amodal segmentation. Our approach is motivated by the “occlusion-free bias” of inpainting models, i.e., the inpainted objects tend to be complete and without occlusions. We reconstruct the occluded regions of an object via inpainting and then apply segmentation, all without additional training or fine-tuning. Experiments on five datasets, three previously unreported, demonstrate the generalizability of our approach. On average, our approach achieves 5.3% more accurate masks in mIoU compared to the publicly available state-of-the-art, pix2gestalt. 

We introduce RGB2Point, an unposed single-view RGB image to a 3D point cloud generation based on Transformer. RGB2Point takes an input image of an object and generates a dense 3D point cloud. Contrary to prior works based on CNN layers and diffusion-denoising approaches, we use pre-trained Transformer layers that are fast and generate high-quality point clouds with consistent quality over available categories. Our generated point clouds demonstrate high quality on a real-world dataset, as evidenced by improved Chamfer distance (51.15%) and Earth Mover’s distance (36.17%) metrics compared to the current state-of the-art. Additionally, our approach shows a better quality on a synthetic dataset, achieving better Chamfer distance (39.26%), Earth Mover’s distance (26.95%), and F-score (47.16%). Moreover, our method produces 63.1% more consistent high-quality results across various object categories compared to prior works. Furthermore, RGB2Point is computationally efficient, requiring only 2.3GB of VRAM to reconstruct a 3D point cloud from a single RGB image, and our implementation generates the results 15,133× faster than a SOTA diffusion-based model.