skip to main content

Title: Application of image processing and convolutional neural networks for flood image classification and semantic segmentation
Deep learning algorithms are exceptionally valuable tools for collecting and analyzing the catastrophic readiness and countless actionable flood data. Convolutional neural networks (CNNs) are one form of deep learning algorithms widely used in computer vision which can be used to study flood images and assign learnable weights to various objects in the image. Here, we leveraged and discussed how connected vision systems can be used to embed cameras, image processing, CNNs, and data connectivity capabilities for flood label detection. We built a training database service of >9000 images (image annotation service) including the image geolocation information by streaming relevant images from social media platforms, Department of Transportation (DOT) 511 traffic cameras, the US Geological Survey (USGS) live river cameras, and images downloaded from search engines. We then developed a new python package called “FloodImageClassifier” to classify and detect objects within the collected flood images. “FloodImageClassifier” includes various CNNs architectures such as YOLOv3 (You look only once version 3), Fast R–CNN (Region-based CNN), Mask R–CNN, SSD MobileNet (Single Shot MultiBox Detector MobileNet), and EfficientDet (Efficient Object Detection) to perform both object detection and segmentation simultaneously. Canny Edge Detection and aspect ratio concepts are also included in the package for flood water more » level estimation and classification. The pipeline is smartly designed to train a large number of images and calculate flood water levels and inundation areas which can be used to identify flood depth, severity, and risk. “FloodImageClassifier” can be embedded with the USGS live river cameras and 511 traffic cameras to monitor river and road flooding conditions and provide early intelligence to emergency response authorities in real-time. « less
Award ID(s):
Publication Date:
Journal Name:
Environmental modelling software
Sponsoring Org:
National Science Foundation
More Like this
  1. Due to the importance of object detection in video analysis and image annotation, it is widely utilized in a number of computer vision tasks such as face recognition, autonomous vehicles, activity recognition, tracking objects and identity verification. Object detection does not only involve classification and identification of objects within images, but also involves localizing and tracing the objects by creating bounding boxes around the objects and labelling them with their respective prediction scores. Here, we leverage and discuss how connected vision systems can be used to embed cameras, image processing, Edge Artificial Intelligence (AI), and data connectivity capabilities for flood label detection. We favored the engineering definition of label detection that a label is a sequence of discrete measurable observations obtained using a capturing device such as web cameras, smart phone, etc. We built a Big Data service of around 1000 images (image annotation service) including the image geolocation information from various flooding events in the Carolinas (USA) with a total of eight different object categories. Our developed platform has several smart AI tools and task configurations that can detect objects’ edges or contours which can be manually adjusted with a threshold setting so as to best segment the image.more »The tool has the ability to train the dataset and predict the labels for large scale datasets which can be used as an object detector to drastically reduce the amount of time spent per object particularly for real-time image-based flood forecasting. This research is funded by the US National Science Foundation (NSF).« less
  2. Skateboarding as a method of transportation has become prevalent, which has increased the occurrence and likelihood of pedestrian–skateboarder collisions and near-collision scenarios in shared-use roadway areas. Collisions between pedestrians and skateboarders can result in significant injury. New approaches are needed to evaluate shared-use areas prone to hazardous pedestrian–skateboarder interactions, and perform real-time, in situ (e.g., on-device) predictions of pedestrian–skateboarder collisions as road conditions vary due to changes in land usage and construction. A mechanism called the Surrogate Safety Measures for skateboarder–pedestrian interaction can be computed to evaluate high-risk conditions on roads and sidewalks using deep learning object detection models. In this paper, we present the first ever skateboarder–pedestrian safety study leveraging deep learning architectures. We view and analyze state of the art deep learning architectures, namely the Faster R-CNN and two variants of the Single Shot Multi-box Detector (SSD) model to select the correct model that best suits two different tasks: automated calculation of Post Encroachment Time (PET) and finding hazardous conflict zones in real-time. We also contribute a new annotated data set that contains skateboarder–pedestrian interactions that has been collected for this study. Both our selected models can detect and classify pedestrians and skateboarders correctly and efficiently. However, duemore »to differences in their architectures and based on the advantages and disadvantages of each model, both models were individually used to perform two different set of tasks. Due to improved accuracy, the Faster R-CNN model was used to automate the calculation of post encroachment time, whereas to determine hazardous regions in real-time, due to its extremely fast inference rate, the Single Shot Multibox MobileNet V1 model was used. An outcome of this work is a model that can be deployed on low-cost, small-footprint mobile and IoT devices at traffic intersections with existing cameras to perform on-device inferencing for in situ Surrogate Safety Measurement (SSM), such as Time-To-Collision (TTC) and Post Encroachment Time (PET). SSM values that exceed a hazard threshold can be published to an Message Queuing Telemetry Transport (MQTT) broker, where messages are received by an intersection traffic signal controller for real-time signal adjustment, thus contributing to state-of-the-art vehicle and pedestrian safety at hazard-prone intersections.« less
  3. Traffic intersections are prime locations for deployment of infrastructure sensors and edge computing nodes to realize the vision of a smart city. It is expected that the needs of a smart city, in regards to traffic and pedestrian traffic systems monitored by cameras/video, can be met by using stateof-the-art artificial-intelligence (AI) based object detectors and trackers. A critical component in designing an effective real-time object detection/tracking pipeline is the understanding of how object density, i.e., the number of objects in a scene, and imageresolution and frame rate influence the performance metrics. This study explores the accuracy and speed metrics with the goal of supporting pipelines that meet the precision and latency needs of a real-time environment. We examine the impact of varying image-resolution, frame rate and object-density on the object detection performance metrics. The experiments on the COSMOS testbed dataset show that varying the frame width from 416 pixels to 832 pixels, and cropping the images to a square resolution, result in the increase in average precision for all object classes. Decreasing the frame rate from 15 fps to 5 fps preserves more than 90% of the highest F1 score achieved for all object classes. The results inform the choicemore »of video preprocessing stages, modifications to established AI-based object detection/tracking methods, and suggest optimal hyper-parameter values. Index Terms—Object Detection, Smart City, Video Resolution, Deep Learning Models.« less
  4. Already known as densely populated areas with land use including housing, transportation, sanitation, utilities and communication, nowadays, cities tend to grow even bigger. Genuine road-user's types are emerging with further technological developments to come. As cities population size escalates, and roads getting congested, government agencies such as Department of Transportation (DOT) through the National Highway Traffic Safety Administration (NHTSA) are in pressing need to perfect their management systems with new efficient technologies. The challenge is to anticipate on never before seen problems, in their effort to save lives and implement sustainable cost-effective management systems. To make things yet more complicated and a bit daunting, self-driving car will be authorized in a close future in crowded major cities where roads are to be shared among pedestrians, cyclists, cars, and trucks. Roads sizes and traffic signaling will need to be constantly adapted accordingly. Counting and classifying turning vehicles and pedestrians at an intersection is an exhausting task and despite traffic monitoring systems use, human interaction is heavily required for counting. Our approach to resolve traffic intersection turning-vehicles counting is less invasive, requires no road dig up or costly installation. Live or recorded videos from already installed camera all over the cities canmore »be used as well as any camera including cellphones. Our system is based on Neural Network and Deep Learning of object detection along computer vision technology and several methods and algorithms. Our approach will work on still images, recorded-videos, real-time live videos and will detect, classify, track and compute moving object velocity and direction using convolution neural network. Created based upon series of algorithms modeled after the human brain, our system uses NVIDIA Video cards with GPU, CUDA, OPENCV and mathematical vectors systems to perform.« less
  5. In recent decades, computer vision has proven remarkably effective in addressing diverse issues in public health, from determining the diagnosis, prognosis, and treatment of diseases in humans to predicting infectious disease outbreaks. Here, we investigate whether convolutional neural networks (CNNs) can also demonstrate effectiveness in classifying the environmental stages of parasites of public health importance and their invertebrate hosts. We used schistosomiasis as a reference model. Schistosomiasis is a debilitating parasitic disease transmitted to humans via snail intermediate hosts. The parasite affects more than 200 million people in tropical and subtropical regions. We trained our CNN, a feed-forward neural network, on a limited dataset of 5,500 images of snails and 5,100 images of cercariae obtained from schistosomiasis transmission sites in the Senegal River Basin, a region in western Africa that is hyper-endemic for the disease. The image set included both images of two snail genera that are relevant to schistosomiasis transmission – that is, Bulinus spp. and Biomphalaria pfeifferi – as well as snail images that are non-component hosts for human schistosomiasis. Cercariae shed from Bi. pfeifferi and Bulinus spp. snails were classified into 11 categories, of which only two, S. haematobium and S. mansoni , are major etiological agentsmore »of human schistosomiasis. The algorithms, trained on 80% of the snail and parasite dataset, achieved 99% and 91% accuracy for snail and parasite classification, respectively, when used on the hold-out validation dataset – a performance comparable to that of experienced parasitologists. The promising results of this proof-of-concept study suggests that this CNN model, and potentially similar replicable models, have the potential to support the classification of snails and parasite of medical importance. In remote field settings where machine learning algorithms can be deployed on cost-effective and widely used mobile devices, such as smartphones, these models can be a valuable complement to laboratory identification by trained technicians. Future efforts must be dedicated to increasing dataset sizes for model training and validation, as well as testing these algorithms in diverse transmission settings and geographies.« less