An official website of the United States government
Automatic satellite-based reconstruction enables large and widespread creation of urban areas. However, satellite imagery is often noisy and incomplete, and is not suitable for reconstructing detailed building facades. We present a machine learning-based inverse procedural modeling method to automatically create synthetic facades from satellite imagery. Our key observation is that building facades exhibit regular, grid-like structures. Hence, we can overcome the low-resolution, noisy, and partial building data obtained from satellite imagery by synthesizing the underlying facade layout. Our method infers regular facade details from satellite-based image-fragments of a building, and applies them to occluded or under-sampled parts of the building, resulting in plausible, crisp facades. Using urban areas from six cities, we compare our approach to several state-of-the-art image completion/in-filling methods and our approach consistently creates better facade images.
more »
« less
This content will become publicly available on June 19, 2026
NeRF-Enhanced Digital Twin for Building Anomaly Inspection Using Unmanned Aerial Systems (UASs)
Building facade inspections are crucial for ensuring structural integrity and occupant safety, traditionally requiring physical access that can be costly and risky. Recent advancements have seen the integration of Unmanned Aerial Systems (UASs) equipped with imaging technologies to facilitate these inspections. However, the primary challenge remains in the effective detection and precise localization of facade anomalies, such as cracks, stains, and specifically, thermal anomalies. This paper investigates the fusing of diverse data sources, specifically thermal infrared (IR) imaging and high-resolution RGB data, to enhance 3D registration of thermal anomalies. Utilizing a Computational Neural Radiance Field (NeRF) approach, it aims to reconstruct detailed 3D models of building facades. This integration improves the accuracy of anomaly detection by fusing the precise camera positions derived from the original RGB data to refine the alignment and visualization of thermal anamolies in both RGB and IR imagery. We utilized COLMAP for camera position estimation and NeRF for 3D registration, employing Structure-from-Motion (SfM) and NeRF methodologies to create detailed and scalable 3D models from 2D IR and RGB images. This approach integrates the precision of camera positioning with advanced 3D reconstruction techniques to enhance the visualization and analysis of building facades. Our method automates the registration and mapping of detected thermal anomalies in 2D images onto the reconstructed 3D models, improving the diagnostics of building facades by enabling precise localization and scaling of these anomalies. The findings demonstrate the potential of our approach to reduce inspection times and enhance the safety of diagnostic procedures through higher accuracy and less invasive methods.
more »
« less
- Award ID(s):
- 2431468
- PAR ID:
- 10651774
- Publisher / Repository:
- Purdue e-Pubs
- Date Published:
- Journal Name:
- CIB Conferences
- Volume:
- 1
- Issue:
- 1
- ISSN:
- 3067-4883
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
This paper introduces an innovative approach to 3D environmental mapping through the integration of a compact, handheld sensor package with a two-stage sensor fusion pipeline. The sensor package, incorporating LiDAR, IMU, RGB, and thermal cameras, enables comprehensive and robust 3D mapping of various environments. By leveraging Simultaneous Localization and Mapping (SLAM) and thermal imaging, our solution offers good performance in conditions where global positioning is unavailable and in visually degraded environments. The sensor package runs a real-time LiDAR-Inertial SLAM algorithm, generating a dense point cloud map that accurately reconstructs the geometric features of the environment. Following the acquisition of that point cloud, we post-process these data by fusing them with images from the RGB and thermal cameras and produce a detailed, color-enriched 3D map that is useful and adaptable to different mission requirements. We demonstrated our system in a variety of scenarios, from indoor to outdoor conditions, and the results showcased the effectiveness and applicability of our sensor package and fusion pipeline. This system can be applied in a wide range of applications, ranging from autonomous navigation to smart agriculture, and has the potential to make a substantial benefit across diverse fields.more » « less
-
The study of non-line-of-sight (NLOS) imaging is growing due to its many potential applications, including rescue operations and pedestrian detection by self-driving cars. However, implementing NLOS imaging on a moving camera remains an open area of research. Existing NLOS imaging methods rely on time-resolved detectors and laser configurations that require precise optical alignment, making it difficult to deploy them in dynamic environments. This work proposes a data-driven approach to NLOS imaging, PathFinder, that can be used with a standard RGB camera mounted on a small, power-constrained mobile robot, such as an aerial drone. Our experimental pipeline is designed to accurately estimate the 2D trajectory of a person who moves in a Manhattan-world environment while remaining hidden from the camera’s fieldof- view. We introduce a novel approach to process a sequence of dynamic successive frames in a line-of-sight (LOS) video using an attention-based neural network that performs inference in real-time. The method also includes a preprocessing selection metric that analyzes images from a moving camera which contain multiple vertical planar surfaces, such as walls and building facades, and extracts planes that return maximum NLOS information. We validate the approach on in-the-wild scenes using a drone for video capture, thus demonstrating low-cost NLOS imaging in dynamic capture environments.more » « less
-
Freshwater mussels are essential parts of our ecosystems to reduce water pollution. As natural bio-filters, they deactivate pollutants such as heavy metals, providing a sustainable method for water decontamination. This project will enable the use of Artificial Intelligence (AI) to monitor mussel behavior, particularly their gaping activity, to use them as bio-indicators for early detection of water contamination. In this paper, we employ advanced 3D reconstruction techniques to create detailed models of mussels to improve the accuracy of AI-based analysis. Specifically, we use a state-of-the-art 3D reconstruction tool, Neural Radiance Fields (NeRF), to create 3D models of mussel valve configurations and behavioral patterns. NeRF enables 3D reconstruction of scenes and objects from a sparse set of 2D images. To capture these images, we developed a data collection system capable of imaging mussels from multiple viewpoints. The system featured a turntable made of foam board with markers around the edges and a designated space in the center for mounting the mussels. The turntable was attached to a servo motor controlled by an ESP32 microcontroller. It rotated in a few degree increments, with the ESP32 camera capturing an image at each step. The images, along with degree information and timestamps, are stored on a Secure Digital (SD) memory card. Several components, such as the camera holder and turntable base, are 3D printed. These images are used to train a NeRF model using the Python-based Nerfstudio framework, and the resulting 3D models were viewed via the Nerfstudio API. The setup was designed to be user-friendly, making it easy for educational outreach engagements and to involve secondary education by replicating and operating 3D reconstructions of their chosen objects. We validated the accessibility and the impact of this platform in a STEM education summer program. A team of high school students from the Juntos Summer Academy at NC State University worked on this platform, gaining hands-on experience in embedded hardware development, basic machine learning principles, and 3D reconstruction from 2D images. We also report on their feedback on the activity.more » « less
-
Recovering rigid registration between successive camera poses lies at the heart of 3D reconstruction, SLAM and visual odometry. Registration relies on the ability to compute discriminative 2D features in successive camera images for determining feature correspondences, which is very challenging in feature-poor environments, i.e. low-texture and/or low-light environments. In this paper, we aim to address the challenge of recovering rigid registration between successive camera poses in feature-poor environments in a Visual Inertial Odometry (VIO) setting. In addition to inertial sensing, we instrument a small aerial robot with an RGBD camera and propose a framework that unifies the incorporation of 3D geometric entities: points, lines, and planes. The tracked 3D geometric entities provide constraints in an Extended Kalman Filtering framework. We show that by directly exploiting 3D geometric entities, we can achieve improved registration. We demonstrate our approach on different texture-poor environments, with some containing only flat texture-less surfaces providing essentially no 2D features for tracking. In addition, we evaluate how the addition of different 3D geometric entities contributes to improved pose estimation by comparing an estimated pose trajectory to a ground truth pose trajectory obtained from a motion capture system. We consider computationally efficient methods for detecting 3D points, lines and planes, since our goal is to implement our approach on small mobile robots, such as drones.more » « less
