Carbon nanotube (CNT) forests are imaged using scanning
electron microscopes (SEMs) that project their multilayered
3D structure into a single 2D image. Image analytics,
particularly instance segmentation is needed to quantify
structural characteristics and to predict correlations between
structural morphology and physical properties. The
inherent complexity of individual CNT structures is further
increased in CNT forests due to density of CNTs, interactions
between CNTs, occlusions, and lack of 3D information
to resolve correspondences when multiple CNTs from
different depths appear to cross in 2D. In this paper, we propose
CNT-NeRF, a generative adversarial network (GAN)
for simultaneous depth layer decomposition and segmentation
of CNT forests in SEM images. The proposed network is
trained using a multi-layer, photo-realistic synthetic dataset
obtained by transferring the style of real CNT images to
physics-based simulation data. Experiments show promising
depth layer decomposition and accurate CNT segmentation
results not only for the front layer but also for the
partially occluded middle and back layers. This achievement
is a significant step towards automated, image-based
CNT forest structure characterization and physical property
prediction.
more »
« less
Self-Supervised Orientation-Guided Deep Network for Segmentation of Carbon Nanotubes in SEM Imagery
Electron microscopy images of carbon nanotube (CNT) forests are difficult to segment due to the long and thin nature of the CNTs; density of the CNT forests resulting in CNTs touching, crossing, and occluding each other; and low signal-to-noise ratio electron microscopy imagery. In addition, due to image complexity, it is not feasible to prepare training segmentation masks. In this paper, we propose CNTSegNet, a dual loss, orientation-guided, self-supervised, deep learning network for CNT forest segmentation in scanning electron microscopy (SEM) images. Our training labels consist of weak segmentation labels produced by intensity thresholding of the raw SEM images and self labels produced by estimating orientation distribution of CNTs in these raw images. The proposed network extends a U-net-like encoder-decoder architecture with a novel two-component loss function. The first component is dice loss computed between the predicted segmentation maps and the weak segmentation labels. The second component is mean squared error (MSE) loss measuring the difference between the orientation histogram of the predicted segmentation map and the original raw image. Weighted sum of these two loss functions is used to train the proposed CNTSegNet network. The dice loss forces the network to perform background-foreground segmentation using local intensity features. The MSE loss guides the network with global orientation features and leads to refined segmentation results. The proposed system needs only a few-shot dataset for training. Thanks to it’s self-supervised nature, it can easily be adapted to new datasets.
more »
« less
- Award ID(s):
- 2026847
- NSF-PAR ID:
- 10441239
- Date Published:
- Journal Name:
- Computer Vision – ECCV 2022 Workshops. ECCV 2022
- Volume:
- 13808
- Page Range / eLocation ID:
- 412-418
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
more » « less
Extracting roads in aerial images has numerous applications in artificial intelligence and multimedia computing, including traffic pattern analysis and parking space planning. Learning deep neural networks, though very successful, demands vast amounts of high-quality annotations, of which acquisition is time-consuming and expensive. In this work, we propose a semi-supervised approach for image-based road extraction where only a small set of labeled images are available for training to address this challenge. We design a pixel-wise contrastive loss to self-supervise the network training to utilize the large corpus of unlabeled images. The key idea is to identify pairs of overlapping image regions (positive) or non-overlapping image regions (negative) and encourage the network to make similar outputs for positive pairs or dissimilar outputs for negative pairs. We also develop a negative sampling strategy to filter false negative samples during the process. An iterative procedure is introduced to apply the network over raw images to generate pseudo-labels, filter and select high-quality labels with the proposed contrastive loss, and re-train the network with the enlarged training dataset. We repeat these iterative steps until convergence. We validate the effectiveness of the proposed methods by performing extensive experiments on the public SpaceNet3 and DeepGlobe Road datasets. Results show that our proposed method achieves state-of-the-art results on public image segmentation benchmarks and significantly outperforms other semi-supervised methods.
-
While the physical properties of carbon nanotubes (CNTs) are often superior to conventional engineering materials, their widespread adoption into many applications is limited by scaling the properties of individual CNTs to macroscale CNT assemblies known as CNT forests. The self-assembly mechanics of CNT forests that determine their morphology and ensemble properties remain poorly understood. Few experimental techniques exist to characterize and observe the growth and self-assembly processes in situ. Here we introduce the use of in-situ scanning electron microscope (SEM) synthesis based on chemical vapor deposition (CVD) processing. In this preliminary report, we share best practices for in-situ SEM CVD processing and initial CNT forest synthesis results. Image analysis techniques are developed to identify and track the movement of catalyst nanoparticles during synthesis conditions. Finally, a perspective is provided in which in-situ SEM observations represent one component of a larger system in which numerical simulation, machine learning, and digital control of experiments reduces the role of humans and human error in the exploration of CNT forest process-structure-property relationships.more » « less
-
In the field of materials science, microscopy is the first and often only accessible method for structural characterization. There is a growing interest in the development of machine learning methods that can automate the analysis and interpretation of microscopy images. Typically training of machine learning models requires large numbers of images with associated structural labels, however, manual labeling of images requires domain knowledge and is prone to human error and subjectivity. To overcome these limitations, we present a semi-supervised transfer learning approach that uses a small number of labeled microscopy images for training and performs as effectively as methods trained on significantly larger image datasets. Specifically, we train an image encoder with unlabeled images using self-supervised learning methods and use that encoder for transfer learning of different downstream image tasks (classification and segmentation) with a minimal number of labeled images for training. We test the transfer learning ability of two self-supervised learning methods: SimCLR and Barlow-Twins on transmission electron microscopy (TEM) images. We demonstrate in detail how this machine learning workflow applied to TEM images of protein nanowires enables automated classification of nanowire morphologies ( e.g. , single nanowires, nanowire bundles, phase separated) as well as segmentation tasks that can serve as groundwork for quantification of nanowire domain sizes and shape analysis. We also extend the application of the machine learning workflow to classification of nanoparticle morphologies and identification of different type of viruses from TEM images.more » « less
-
Medical image segmentation is one of the most challenging tasks in medical image analysis and widely developed for many clinical applications. While deep learning-based approaches have achieved impressive performance in semantic segmentation, they are limited to pixel-wise settings with imbalanced-class data problems and weak boundary object segmentation in medical images. In this paper, we tackle those limitations by developing a new two-branch deep network architecture which takes both higher level features and lower level features into account. The first branch extracts higher level feature as region information by a common encoder-decoder network structure such as Unet and FCN, whereas the second branch focuses on lower level features as support information around the boundary and processes in parallel to the first branch. Our key contribution is the second branch named Narrow Band Active Contour (NB-AC) attention model which treats the object contour as a hyperplane and all data inside a narrow band as support information that influences the position and orientation of the hyperplane. Our proposed NB-AC attention model incorporates the contour length with the region energy involving a fixed-width band around the curve or surface. The proposed network loss contains two fitting terms: (i) a high level feature (i.e., region) fitting term from the first branch; (ii) a lower level feature (i.e., contour) fitting term from the second branch including the (ii1) length of the object contour and (ii2) regional energy functional formed by the homogeneity criterion of both the inner band and outer band neighboring the evolving curve or surface. The proposed NB-AC loss can be incorporated into both 2D and 3D deep network architectures. The proposed network has been evaluated on different challenging medical image datasets, including DRIVE, iSeg17, MRBrainS18 and Brats18. The experimental results have shown that the proposed NB-AC loss outperforms other mainstream loss functions: Cross Entropy, Dice, Focal on two common segmentation frameworks Unet and FCN. Our 3D network which is built upon the proposed NB-AC loss and 3DUnet framework achieved state-of-the-art results on multiple volumetric datasets.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
