skip to main content


Title: Enhancing Deep Neural Network-Based Trajectory Prediction: Fine-Tuning and Inherent Movement-Driven Post-Processing
As a proactive means of preventing struck-by accidents in construction, many studies have presented proximity monitoring applications using wireless sensors (e.g., RFID, UWB, and GPS) or computer vision methods. Most prior research has emphasized proximity detection rather than prediction. However, prediction can be more effective and important for contact-driven accident prevention, particularly given that the sooner workers (e.g., equipment operators and workers on foot) are informed of their proximity to each other, the more likely they are to avoid the impending collision. In earlier studies, the authors presented a trajectory prediction method leveraging a deep neural network to examine the feasibility of proximity prediction in real-world applications. In this study, we enhance the existing trajectory prediction accuracy. Specifically, we improve the trajectory prediction model by tuning its pre-trained weight parameters with construction data. Moreover, inherent movement-driven post-processing algorithm is developed to refine the trajectory prediction of a target in accordance with its inherent movement patterns such as the final position, predominant direction, and average velocity. In a test on real-site operations data, the proposed approach demonstrates the improvement in accuracy: for 5.28 seconds’ prediction, it achieves 0.39 meter average displacement error, improved by 51.43% as compared with the previous one (0.84 meters). The improved trajectory prediction method can support to predict potential contact-driven hazards in advance, which can allow for prompt feedback (e.g., visible, acoustic, and vibration alarms) to equipment operators and workers on foot. The proactive intervention can lead the workers to take prompt evasive action, thereby reducing the chance of an impending collision.  more » « less
Award ID(s):
1734266
NSF-PAR ID:
10201431
Author(s) / Creator(s):
; ; ;
Editor(s):
El Asmar, Mounir; Grau, David; Tang, Pingbo
Date Published:
Journal Name:
Construction Research Congress 2020
Page Range / eLocation ID:
67 to 75
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Cellular service carriers often employ reactive strategies to assist customers who experience non-outage related individual service degradation issues (e.g., service performance degradations that do not impact customers at scale and are likely caused by network provisioning issues for individual devices). Customers need to contact customer care to request assistance before these issues are resolved. This paper presents our experience with PACE (ProActive customer CarE), a novel, proactive system that monitors, troubleshoots and resolves individual service issues, without having to rely on customers to first contact customer care for assistance. PACE seeks to improve customer experience and care operation efficiency by automatically detecting individual (non-outage related) service issues, prioritizing repair actions by predicting customers who are likely to contact care to report their issues, and proactively triggering actions to resolve these issues. We develop three machine learning-based prediction models, and implement a fully automated system that integrates these prediction models and takes resolution actions for individual customers.We conduct a large-scale trace-driven evaluation using real-world data collected from a major cellular carrier in the US, and demonstrate that PACE is able to predict customers who are likely to contact care due to non-outage related individual service issues with high accuracy. We further deploy PACE into this cellular carrier network. Our field trial results show that PACE is effective in proactively resolving non-outage related individual customer service issues, improving customer experience, and reducing the need for customers to report their service issues. 
    more » « less
  2. Robots and humans closely working together within dynamic environments must be able to continuously look ahead and identify potential collisions within their ever-changing environment. To enable the robot to act upon such situational awareness, its controller requires an iterative collision detection capability that will allow for computationally efficient Proactive Adaptive Collaboration Intelligence (PACI) to ensure safe interactions. In this paper, an algorithm is developed to evaluate a robot’s trajectory, evaluate the dynamic environment that the robot operates in, and predict collisions between the robot and dynamic obstacles in its environment. This algorithm takes as input the joint motion data of predefined robot execution plans and constructs a sweep of the robot’s instantaneous poses throughout time. The sweep models the trajectory as a point cloud containing all locations occupied by the robot and the time at which they will be occupied. To reduce the computational burden, Coons patches are leveraged to approximate the robot’s instantaneous poses. In parallel, the algorithm creates a similar sweep to model any human(s) and other obstacles being tracked in the operating environment. Overlaying temporal mapping of the sweeps reveals anticipated collisions that will occur if the robot-human do not proactively modify their motion. The algorithm is designed to feed into a segmentation and switching logic framework and provide real-time proactive-n-reactive behavior for different levels of human-robot interactions, while maintaining safety and production efficiency. To evaluate the predictive collision detection approach, multiple test cases are presented to quantify the computational speed and accuracy in predicting collisions. 
    more » « less
  3. null (Ed.)
    In proportion to the immense construction of spatial structures is the emergence of catastrophes related to structural damages (e.g. loose connections), thus rendering personal injury and property loss. It is therefore essential to detect spatial bolt looseness. Current methods for detecting spatial bolt looseness mostly focus on contact-type measurement, which may not be practical in some cases. Thus, inspired by the sound-based human diagnostic approach, we develop a novel percussion method using the Mel-frequency cepstral coefficient and the memory-augmented neural network in this article. In comparison with current investigations, the main contribution of this article is the detection of multi-bolt looseness for the first time with higher accuracy than prior methods. In particular, in terms of new data obtained via similar joints, the memory-augmented neural network can help avoid inefficient relearn and assimilate new data to provide accurate prediction with only a few data samples, which effectively improves the robustness of detection. Furthermore, percussion was implemented with a robotic arm instead of manual operation, which preliminarily explores the potential of implementing automation applications in real industries. Finally, experimental results demonstrate the effectiveness of the proposed method, which can guide future development of cyber-physics systems for structural health detection. 
    more » « less
  4. Skateboarding as a method of transportation has become prevalent, which has increased the occurrence and likelihood of pedestrian–skateboarder collisions and near-collision scenarios in shared-use roadway areas. Collisions between pedestrians and skateboarders can result in significant injury. New approaches are needed to evaluate shared-use areas prone to hazardous pedestrian–skateboarder interactions, and perform real-time, in situ (e.g., on-device) predictions of pedestrian–skateboarder collisions as road conditions vary due to changes in land usage and construction. A mechanism called the Surrogate Safety Measures for skateboarder–pedestrian interaction can be computed to evaluate high-risk conditions on roads and sidewalks using deep learning object detection models. In this paper, we present the first ever skateboarder–pedestrian safety study leveraging deep learning architectures. We view and analyze state of the art deep learning architectures, namely the Faster R-CNN and two variants of the Single Shot Multi-box Detector (SSD) model to select the correct model that best suits two different tasks: automated calculation of Post Encroachment Time (PET) and finding hazardous conflict zones in real-time. We also contribute a new annotated data set that contains skateboarder–pedestrian interactions that has been collected for this study. Both our selected models can detect and classify pedestrians and skateboarders correctly and efficiently. However, due to differences in their architectures and based on the advantages and disadvantages of each model, both models were individually used to perform two different set of tasks. Due to improved accuracy, the Faster R-CNN model was used to automate the calculation of post encroachment time, whereas to determine hazardous regions in real-time, due to its extremely fast inference rate, the Single Shot Multibox MobileNet V1 model was used. An outcome of this work is a model that can be deployed on low-cost, small-footprint mobile and IoT devices at traffic intersections with existing cameras to perform on-device inferencing for in situ Surrogate Safety Measurement (SSM), such as Time-To-Collision (TTC) and Post Encroachment Time (PET). SSM values that exceed a hazard threshold can be published to an Message Queuing Telemetry Transport (MQTT) broker, where messages are received by an intersection traffic signal controller for real-time signal adjustment, thus contributing to state-of-the-art vehicle and pedestrian safety at hazard-prone intersections. 
    more » « less
  5. We explore the possibility of using a single monocular camera to forecast the time to collision between a suitcase-shaped robot being pushed by its user and other nearby pedestrians. We develop a purely image-based deep learning approach that directly estimates the time to collision without the need of relying on explicit geometric depth estimates or velocity information to predict future collisions. While previous work has focused on detecting immediate collision in the context of navigating Unmanned Aerial Vehicles, the detection was limited to a binary variable (i.e., collision or no collision). We propose a more fine-grained approach to collision forecasting by predicting the exact time to collision in terms of milliseconds, which is more helpful for collision avoidance in the context of dynamic path planning. To evaluate our method, we have collected a novel dataset of over 13,000 indoor video segments each showing a trajectory of at least one person ending in a close proximity (a near collision) with the camera mounted on a mobile suitcase-shaped platform. Using this dataset, we do extensive experimentation on different temporal windows as input using an exhaustive list of state-of-the-art convolutional neural networks (CNNs). Our results show that our proposed multi-stream CNN is the best model for predicting time to near-collision. The average prediction error of our time to near-collision is 0.75 seconds across the test videos. The project webpage can be found at https://aashi7.github.io/NearCollision.html. 
    more » « less