More Like this

1. Image and Text Feature Based Multimodal Learning for Multi-Label Classification of Radiology Images in Biomedical Literature [Image and Text Feature Based Multimodal Learning for Multi-Label Classification of Radiology Images in Biomedical Literature]

   https://doi.org/10.5220/0012438400003657  

   Hasan, Md; Jani, Md; Rahman, Md (January 2024, SCITEPRESS - Science and Technology Publications)

   Biomedical images are crucial for diagnosing and planning treatments, as well as advancing scientific understanding of various ailments. To effectively highlight regions of interest (RoIs) and convey medical concepts, annotation markers like arrows, letters, or symbols are employed. However, annotating these images with appropriate medical labels poses a significant challenge. In this study, we propose a framework that leverages multimodal input features, including text/label features and visual features, to facilitate accurate annotation of biomedical images with multiple labels. Our approach integrates state-of-the-art models such as ResNet50 and Vision Transformers (ViT) to extract informative features from the images. Additionally, we employ Generative Pre-trained Distilled-GPT2 (Transformer based Natural Language Processing architecture) to extract textual features, leveraging their natural language understanding capabilities. This combination of image and text modalities allows for a more comprehensive representation of the biomedical data, leading to improved annotation accuracy. By combining the features extracted from both image and text modalities, we trained a simplified Convolutional Neural Network (CNN) based multi-classifier to learn the image-text relations and predict multi-labels for multi-modal radiology images. We used ImageCLEFmedical 2022 and 2023 datasets to demonstrate the effectiveness of our framework. This dataset likely contains a diverse range of biomedical images, enabling the evaluation of the framework’s performance under realistic conditions. We have achieved promising results with the F1 score of 0.508. Our proposed framework exhibits potential performance in annotating biomedical images with multiple labels, contributing to improved image understanding and analysis in the medical image processing domain. 

   more » « less
2. Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Models

   Chu, Tianzhe Chu; Tong, Shengbang; Ding, Tianjiao; Dai, Xili; Haeffele, Benjamin; Vidal, René; Ma, Yi (May 2024, International Conference on Learning Representations (ICLR))

   The advent of large pre-trained models has brought about a paradigm shift in both visual representation learning and natural language processing. However, clustering unlabeled images, as a fundamental and classic machine learning problem, still lacks an effective solution, particularly for large-scale datasets. In this paper, we propose a novel image clustering pipeline that leverages the powerful feature representation of large pre-trained models such as CLIP and cluster images effectively and efficiently at scale. We first developed a novel algorithm to estimate the number of clusters in a given dataset. We then show that the pre-trained features are significantly more structured by further optimizing the rate reduction objective. The resulting features may significantly improve the clustering accuracy, e.g., from 57\% to 66\% on ImageNet-1k. Furthermore, by leveraging CLIP's multimodality bridge between image and text, we develop a simple yet effective self-labeling algorithm that produces meaningful captions for the clusters. Through extensive experiments, we show that our pipeline works well on standard datasets such as CIFAR-10, CIFAR-100, and ImageNet-1k. It also extends to datasets that are not curated for clustering, such as LAION-Aesthetics and WikiArts. We released the code in https://github.com/LeslieTrue/CPP 

   more » « less
3. Increasing the Trustworthiness of Deep Neural Networks via Accuracy Monitoring

   Zhihui Shao, Jianyi Yang (January 2020, Workshop on Artificial Intelligence Safety 2020 (co-located with IJCAI-PRICAI 2020))

   Inference accuracy of deep neural networks (DNNs) is a crucial performance metric, but can vary greatly in practice subject to actual test datasets and is typically unknown due to the lack of ground truth labels. This has raised significant concerns with trustworthiness of DNNs, especially in safety-critical applications. In this paper, we address trustworthiness of DNNs by using post-hoc processing to monitor the true inference accuracy on a user’s dataset. Concretely, we propose a neural network-based accuracy monitor model, which only takes the deployed DNN’s softmax probability output as its input and directly predicts if the DNN’s prediction result is correct or not, thus leading to an estimate of the true inference accuracy. The accuracy monitor model can be pre-trained on a dataset relevant to the target application of interest, and only needs to actively label a small portion (1% in our experiments) of the user’s dataset for model transfer. For estimation robustness, we further employ an ensemble of monitor models based on the Monte-Carlo dropout method. We evaluate our approach on different deployed DNN models for image classification and traffic sign detection over multiple datasets (including adversarial samples). The result shows that our accuracy monitor model provides a close-to-true accuracy estimation and outperforms the existing baseline methods. 

   more » « less
4. Image-based PDF Malware Detection Using Pre-trained Deep Neural Networks

   https://doi.org/10.1109/ISDFS60797.2024.10527343  

   Nichols, Tyler; Zemlanicky, Jack; Luo, Zhirui; Li, Qingqing; Zheng, Jun (April 2024, 2024 12th International Symposium on Digital Forensics and Security (ISDFS))

   PDF is a popular document file format with a flexible file structure that can embed diverse types of content, including images and JavaScript code. However, these features make it a favored vehicle for malware attackers. In this paper, we propose an image-based PDF malware detection method that utilizes pre-trained deep neural networks (DNNs). Specifically, we convert PDF files into fixed-size grayscale images using an image visualization technique. These images are then fed into pre-trained DNN models to classify them as benign or malicious. We investigated four classical pre-trained DNN models in our study. We evaluated the performance of the proposed method using the publicly available Contagio PDF malware dataset. Our results demonstrate that MobileNetv3 achieves the best detection performance with an accuracy of 0.9969 and exhibits low computational complexity, making it a promising solution for image-based PDF malware detection. 

   more » « less
5. Machine learning for automated classification of lung collagen in a urethane-induced lung injury mouse model

   https://doi.org/10.1364/BOE.527972  

   Alnafisah, Khalid_Hamad; Ranjan, Amit; Sahu, Sushant_P; Chen, Jianhua; Alhejji, Sarah_Mohammad; Noël, Alexandra; Gartia, Manas_Ranjan; Mukhopadhyay, Supratik (September 2024, Biomedical Optics Express)

   Dysregulation of lung tissue collagen level plays a vital role in understanding how lung diseases progress. However, traditional scoring methods rely on manual histopathological examination introducing subjectivity and inconsistency into the assessment process. These methods are further hampered by inter-observer variability, lack of quantification, and their time-consuming nature. To mitigate these drawbacks, we propose a machine learning-driven framework for automated scoring of lung collagen content. Our study begins with the collection of a lung slide image dataset from adult female mice using second harmonic generation (SHG) microscopy. In our proposed approach, first, we manually extracted features based on the 46 statistical parameters of fibrillar collagen. Subsequently, we pre-processed the images and utilized a pre-trained VGG16 model to uncover hidden features from pre-processed images. We then combined both image and statistical features to train various machine learning and deep neural network models for classification tasks. We employed advanced unsupervised techniques like K-means, principal component analysis (PCA), t-distributed stochastic neighbour embedding (t-SNE), and uniform manifold approximation and projection (UMAP) to conduct thorough image analysis for lung collagen content. Also, the evaluation of the trained models using the collagen data includes both binary and multi-label classification to predict lung cancer in a urethane-induced mouse model. Experimental validation of our proposed approach demonstrates promising results. We obtained an average accuracy of 83% and an area under the receiver operating characteristic curve (ROC AUC) values of 0.96 through the use of a support vector machine (SVM) model for binary categorization tasks. For multi-label classification tasks, to quantify the structural alteration of collagen, we attained an average accuracy of 73% and ROC AUC values of 1.0, 0.38, 0.95, and 0.86 for control, baseline, treatment_1, and treatment_2 groups, respectively. Our findings provide significant potential for enhancing diagnostic accuracy, understanding disease mechanisms, and improving clinical practice using machine learning and deep learning models. 

   more » « less