More Like this

1. Cultural and Geographical Influences on Image Translatability of Words across Languages

   https://doi.org/10.18653/v1/2021.naacl-main.19  

   Khani, Nikzad; Tourni, Isidora; Rasooli, Mohammad Sadegh; Callison-Burch, Chris; Wijaya, Derry Tanti (January 2021, The 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies)

   null (Ed.)

   Neural Machine Translation (NMT) models have been observed to produce poor translations when there are few/no parallel sentences to train the models. In the absence of parallel data, several approaches have turned to the use of images to learn translations. Since images of words, e.g., horse may be unchanged across languages, translations can be identified via images associated with words in different languages that have a high degree of visual similarity. However, translating via images has been shown to improve upon text-only models only marginally. To better understand when images are useful for translation, we study image translatability of words, which we define as the translatability of words via images, by measuring intra- and inter-cluster similarities of image representations of words that are translations of each other. We find that images of words are not always invariant across languages, and that language pairs with shared culture, meaning having either a common language family, ethnicity or religion, have improved image translatability (i.e., have more similar images for similar words) compared to its converse, regardless of their geographic proximity. In addition, in line with previous works that show images help more in translating concrete words, we found that concrete words have improved image translatability compared to abstract ones. 

   more » « less
2. Automated, high-throughput image calibration for parallel-laser photogrammetry

   https://doi.org/10.1007/s42991-021-00174-7  

   Richardson, Jack L.; Levy, Emily J.; Ranjithkumar, Riddhi; Yang, Huichun; Monson, Eric; Cronin, Arthur; Galbany, Jordi; Robbins, Martha M.; Alberts, Susan C.; Reeves, Mark E.; et al (March 2022, Mammalian Biology)

   Parallel-laser photogrammetry is growing in popularity as a way to collect non-invasive body size data from wild mammals. Despite its many appeals, this method requires researchers to hand-measure (i) the pixel distance between the parallel laser spots (inter-laser distance) to produce a scale within the image, and (ii) the pixel distance between the study subject’s body landmarks (inter-landmark distance). This manual effort is time-consuming and introduces human error: a researcher measuring the same image twice will rarely return the same values both times (resulting in within-observer error), as is also the case when two researchers measure the same image (resulting in between-observer error). Here, we present two independent methods that automate the inter-laser distance measurement of parallel-laser photogrammetry images. One method uses machine learning and image processing techniques in Python, and the other uses image processing techniques in ImageJ. Both of these methods reduce labor and increase precision without sacrificing accuracy. We first introduce the workflow of the two methods. Then, using two parallel-laser datasets of wild mountain gorilla and wild savannah baboon images, we validate the precision of these two automated methods relative to manual measurements and to each other. We also estimate the reduction of variation in final body size estimates in centimeters when adopting these automated methods, as these methods have no human error. Finally, we highlight the strengths of each method, suggest best practices for adopting either of them, and propose future directions for the automation of parallel-laser photogrammetry data. 

   more » « less
3. JECL: Joint Embedding and Cluster Learning for Image-Text Pairs

   https://doi.org/10.1109/ICPR48806.2021.9412667  

   Yang, Sean T.; Huang, Kuan-Hao; Howe, Bill (January 2021, 2020 25th International Conference on Pattern Recognition (ICPR))

   null (Ed.)

   We propose JECL, a method for clustering image-caption pairs by training parallel encoders with regularized clustering and alignment objectives, simultaneously learning both representations and cluster assignments. These image-caption pairs arise frequently in high-value applications where structured training data is expensive to produce, but free-text descriptions are common. JECL trains by minimizing the Kullback-Leibler divergence between the distribution of the images and text to that of a combined joint target distribution and optimizing the Jensen-Shannon divergence between the soft cluster assignments of the images and text. Regularizers are also applied to JECL to prevent trivial solutions. Experiments show that JECL outperforms both single-view and multi-view methods on large benchmark image-caption datasets, and is remarkably robust to missing captions and varying data sizes. 

   more » « less
4. Comparing Deep Learning Approaches for Understanding Genotype × Phenotype Interactions in Biomass Sorghum

   https://doi.org/10.3389/frai.2022.872858  

   Zhang, Zeyu; Pope, Madison; Shakoor, Nadia; Pless, Robert; Mockler, Todd C.; Stylianou, Abby (July 2022, Frontiers in Artificial Intelligence)

   We explore the use of deep convolutional neural networks (CNNs) trained on overhead imagery of biomass sorghum to ascertain the relationship between single nucleotide polymorphisms (SNPs), or groups of related SNPs, and the phenotypes they control. We consider both CNNs trained explicitly on the classification task of predicting whether an image shows a plant with a reference or alternate version of various SNPs as well as CNNs trained to create data-driven features based on learning features so that images from the same plot are more similar than images from different plots, and then using the features this network learns for genetic marker classification. We characterize how efficient both approaches are at predicting the presence or absence of a genetic markers, and visualize what parts of the images are most important for those predictions. We find that the data-driven approaches give somewhat higher prediction performance, but have visualizations that are harder to interpret; and we give suggestions of potential future machine learning research and discuss the possibilities of using this approach to uncover unknown genotype × phenotype relationships. 

   more » « less
5. SRNet: A spatial-relationship aware point-set classification method for multiplexed pathology images.

   Li, Y. (July 2021, In Proceedings of DeepSpatial’21: 2nd ACM SIGKDD Workshop on Deep Learning for Spatiotemporal Data, Applications, and Systems)

   Point-set classification for multiplexed pathology images aims to distinguish between the spatial configurations of cells within multiplexed immuno-fluorescence (mIF) images of different diseases. This problem is important towards aiding pathologists in diag- nosing diseases (e.g., chronic pancreatitis and pancreatic ductal adenocarcinoma). This problem is challenging because crucial spa- tial relationships are implicit in point sets and the non-uniform distribution of points makes the relationships complex. Manual morphologic or cell-count based methods, the conventional clinical approach for studying spatial patterns within mIF images, is limited by inter-observer variability. The current deep neural network methods for point sets (e.g., PointNet) are limited in learning the representation of implicit spatial relationships between categorical points. To overcome the limitation, we propose a new deep neural network (DNN) architecture, namely spatial-relationship aware neural networks (SRNet), with a novel design of representation learning layers. Experimental results with a University of Michigan mIF dataset show that the proposed method significantly outperforms the competing DNN methods, by 80%, reaching 95% accuracy. 

   more » « less