skip to main content


Title: Enabling human–infrastructure interfaces for inspection using augmented reality
Decaying infrastructure maintenance cost allocation depends heavily on accurate and safe inspection in the field. New tools to conduct inspections can assist in prioritizing investments in maintenance and repairs. The industrial revolution termed as “Industry 4.0” is based on the intelligence of machines working with humans in a collaborative workspace. Contrarily, infrastructure management has relied on the human for making day-to-day decisions. New emerging technologies can assist during infrastructure inspections, to quantify structural condition with more objective data. However, today’s owners agree in trusting the inspector’s decision in the field over data collected with sensors. If data collected in the field is accessible during the inspections, the inspector decisions can be improved with sensors. New research opportunities in the human–infrastructure interface would allow researchers to improve the human awareness of their surrounding environment during inspections. This article studies the role of Augmented Reality (AR) technology as a tool to increase human awareness of infrastructure in their inspection work. The domains of interest of this research include both infrastructure inspections (emphasis on the collection of data of structures to inform management decisions) and emergency management (focus on the data collection of the environment to inform human actions). This article describes the use of a head-mounted device to access real-time data and information during their field inspection. The authors leverage the use of low-cost smart sensors and QR code scanners integrated with Augmented Reality applications for augmented human interface with the physical environment. This article presents a novel interface architecture for developing Augmented Reality–enabled inspection to assist the inspector’s workflow in conducting infrastructure inspection works with two new applications and summarizes the results from various experiments. The main contributions of this work to computer-aided community are enabling inspectors to visualize data files from database and real-time data access using an Augmented Reality environment.  more » « less
Award ID(s):
2024520
NSF-PAR ID:
10348626
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
Structural Health Monitoring
Volume:
20
Issue:
4
ISSN:
1475-9217
Page Range / eLocation ID:
1980 to 1996
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract This article represents a systematic effort to answer the question, What are archaeology’s most important scientific challenges? Starting with a crowd-sourced query directed broadly to the professional community of archaeologists, the authors augmented, prioritized, and refined the responses during a two-day workshop focused specifically on this question. The resulting 25 “grand challenges” focus on dynamic cultural processes and the operation of coupled human and natural systems. We organize these challenges into five topics: (1) emergence, communities, and complexity; (2) resilience, persistence, transformation, and collapse; (3) movement, mobility, and migration; (4) cognition, behavior, and identity; and (5) human-environment interactions. A discussion and a brief list of references accompany each question. An important goal in identifying these challenges is to inform decisions on infrastructure investments for archaeology. Our premise is that the highest priority investments should enable us to address the most important questions. Addressing many of these challenges will require both sophisticated modeling and large-scale synthetic research that are only now becoming possible. Although new archaeological fieldwork will be essential, the greatest pay off will derive from investments that provide sophisticated research access to the explosion in systematically collected archaeological data that has occurred over the last several decades. 
    more » « less
  2. Abstract In‐field visual inspections have inherent challenges associated with humans such as low accuracy, excessive cost and time, and safety. To overcome these barriers, researchers and industry leaders have developed image‐based methods for automatic structural crack detection. More recently, researchers have proposed using augmented reality (AR) to interface human visual inspection with automatic image‐based crack detection. However, to date, AR crack detection is limited because: (1) it is not available in real time and (2) it requires an external processing device. This paper describes a new AR methodology that addresses both problems enabling a standalone real‐time crack detection system for field inspection. A Canny algorithm is transformed into the single‐dimensional mathematical environment of the AR headset digital platform. Then, the algorithm is simplified based on the limited headset processing capacity toward lower processing time. The test of the AR crack‐detection method eliminates AR image‐processing dependence on external processors and has practical real‐time image‐processing. 
    more » « less
  3. Abstract

    ROV operations are mainly performed via a traditional control kiosk and limited data feedback methods, such as the use of joysticks and camera view displays equipped on a surface vessel. This traditional setup requires significant personnel on board (POB) time and imposes high requirements for personnel training. This paper proposes a virtual reality (VR) based haptic-visual ROV teleoperation system that can substantially simplify ROV teleoperation and enhance the remote operator's situational awareness.

    This study leverages the recent development in Mixed Reality (MR) technologies, sensory augmentation, sensing technologies, and closed-loop control, to visualize and render complex underwater environmental data in an intuitive and immersive way. The raw sensor data will be processed with physics engine systems and rendered as a high-fidelity digital twin model in game engines. Certain features will be visualized and displayed via the VR headset, whereas others will be manifested as haptic and tactile cues via our haptic feedback systems. We applied a simulation approach to test the developed system.

    With our developed system, a high-fidelity subsea environment is reconstructed based on the sensor data collected from an ROV including the bathymetric, hydrodynamic, visual, and vehicle navigational measurements. Specifically, the vehicle is equipped with a navigation sensor system for real-time state estimation, an acoustic Doppler current profiler for far-field flow measurement, and a bio-inspired artificial literal-line hydrodynamic sensor system for near-field small-scale hydrodynamics. Optimized game engine rendering algorithms then visualize key environmental features as augmented user interface elements in a VR headset, such as color-coded vectors, to indicate the environmental impact on the performance and function of the ROV. In addition, augmenting environmental feedback such as hydrodynamic forces are translated into patterned haptic stimuli via a haptic suit for indicating drift-inducing flows in the near field. A pilot case study was performed to verify the feasibility and effectiveness of the system design in a series of simulated ROV operation tasks.

    ROVs are widely used in subsea exploration and intervention tasks, playing a critical role in offshore inspection, installation, and maintenance activities. The innovative ROV teleoperation feedback and control system will lower the barrier for ROV pilot jobs.

     
    more » « less
  4. Puerto Rico is exposed to multiple hazards including hurricanes, earthquakes, and floods. The Resilient Infrastructure and Sustainability Education Undergraduate Program (RISE-UP) at the University of Puerto Rico aims to introduce students to interdisciplinary problem-solving related to real challenges, especially those associated with the occurrence of natural disasters. The objective of this work is to share our experience with experiential learning related to structural engineering. The lessons learned from this experience, from the student ́s perspective, could encourage faculty members to develop similar undertakings in their programs and students to participate when opportunities arise. During the 2019 fall semester, we enrolled in a course which covered the relationship between design and natural disasters, with an emphasis on rapid response to recover during the aftermath. The course combined lectures and in-class exercises on basic structural analysis, classifications of structures and the use of the FEMA Rapid Visual Screening (P-154) form. This was complemented with field visits of structures affected by Hurricane Maria where we developed several case studies. From December of 2019 to February 2020, Puerto Rico suffered an earthquake swarm reaching magnitudes as high as 6.4, which caused structural damages throughout the South West of the island. Following these events, we were able to use the training acquired during our course in a real-life, post-disaster situation. At the University, we participated in visual inspection brigades, where we aided professional engineers and faculty members in data collection and categorizing building damages. Our involvement helped streamline efforts as we provided additional support in report writing and organization of the data collected using GIS and other tools. The results of the visual inspections indicated that in many cases pre- existing conditions were aggravated by the earthquakes. Furthermore, we also witnessed firsthand the complexities of assessing infrastructure damage during and following high seismic activity. This experience enhanced our awareness of the significance of our profession in ensuring the safety of others both immediately after an earthquake and in the face of future disasters. 
    more » « less
  5. null (Ed.)
    Corrosion on steel bridge members is one of the most important bridge deficiencies that must be carefully monitored by inspectors. Human visual inspection is typically conducted first, and additional measures such as tapping bolts and measuring section losses can be used to assess the level of corrosion. This process becomes a challenge when some of the connections are placed in a location where inspectors have to climb up or down the steel members. To assist this inspection process, we developed a computervision based Unmanned Aerial Vehicle (UAV) system for monitoring the health of critical steel bridge connections (bolts, rivets, and pins). We used a UAV to collect images from a steel truss bridge. Then we fed the collected datasets into an instance level segmentation model using a region-based convolutional neural network to train characteristics of corrosion shown at steel connections with sets of labeled image data. The segmentation model identified locations of the connections in images and efficiently detected the members with corrosion on them. We evaluated the model based on how precisely it can detect rivets, bolts, pins, and corrosion damage on these members. The results showed robustness and practicality of our system which can also provide useful health information to bridge owners for future maintenance. These collected image data can be used to quantitatively track temporal changes and to monitor progression of damage in aging steel structures. Furthermore, the system can also assist inspectors in making decisions for further detailed inspections. 
    more » « less