Title: A Study of Why We Need to Reassess Full Reference Image Quality Assessment with Medical Images
Abstract Image quality assessment (IQA) is indispensable in clinical practice to ensure high standards, as well as in the development stage of machine learning algorithms that operate on medical images. The popular full reference (FR) IQA measures PSNR and SSIM are known and tested for working successfully in many natural imaging tasks, but discrepancies in medical scenarios have been reported in the literature, highlighting the gap between development and actual clinical application. Such inconsistencies are not surprising, as medical images have very different properties than natural images, and PSNR and SSIM have neither been targeted nor properly tested for medical images. This may cause unforeseen problems in clinical applications due to wrong judgement of novel methods. This paper provides a structured and comprehensive overview of examples where PSNR and SSIM prove to be unsuitable for the assessment of novel algorithms using different kinds of medical images, including real-world MRI, CT, OCT, X-Ray, digital pathology and photoacoustic imaging data. Therefore, improvement is urgently needed in particular in this era of AI to increase reliability and explainability in machine learning for medical imaging and beyond. Lastly, we will provide ideas for future research as well as suggest guidelines for the usage of FR-IQA measures applied to medical images.  more » « less
Award ID(s):
2208294
PAR ID:
10627392
Author(s) / Creator(s):
; ; ; ; ; ; ; ; ; ; ; ;
Corporate Creator(s):
Publisher / Repository:
Springer
Date Published:
Journal Name:
Journal of Imaging Informatics in Medicine
ISSN:
2948-2933
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Yang, DN; Xie, X; Tseng, VS; Pei, J; Huang, JW; Lin, JCW (Ed.)
    Extensive research in Medical Imaging aims to uncover critical diagnostic features in patients, with AI-driven medical diagnosis relying on sophisticated machine learning and deep learning models to analyze, detect, and identify diseases from medical images. Despite the remarkable accuracy of these models under normal conditions, they grapple with trustworthiness issues, where their output could be manipulated by adversaries who introduce strategic perturbations to the input images. Furthermore, the scarcity of publicly available medical images, constituting a bottleneck for reliable training, has led contemporary algorithms to depend on pretrained models grounded on a large set of natural images—a practice referred to as transfer learning. However, a significant domain discrepancy exists between natural and medical images, which causes AI models resulting from transfer learning to exhibit heightened vulnerability to adversarial attacks. This paper proposes a domain assimilation approach that introduces texture and color adaptation into transfer learning, followed by a texture preservation component to suppress undesired distortion. We systematically analyze the performance of transfer learning in the face of various adversarial attacks under different data modalities, with the overarching goal of fortifying the model’s robustness and security in medical imaging tasks. The results demonstrate high effectiveness in reducing attack efficacy, contributing toward more trustworthy transfer learning in biomedical applications. 
    more » « less
  2. Yang, DN; Xie, X; Tseng, VS; Pei, J; Huang, JW; Lin, JCW (Ed.)
    Extensive research in Medical Imaging aims to uncover critical diagnostic features in patients, with AI-driven medical diagnosis relying on sophisticated machine learning and deep learning models to analyze, detect, and identify diseases from medical images. Despite the remarkable accuracy of these models under normal conditions, they grapple with trustworthiness issues, where their output could be manipulated by adversaries who introduce strategic perturbations to the input images. Furthermore, the scarcity of publicly available medical images, constituting a bottleneck for reliable training, has led contemporary algorithms to depend on pretrained models grounded on a large set of natural images—a practice referred to as transfer learning. However, a significant domain discrepancy exists between natural and medical images, which causes AI models resulting from transfer learning to exhibit heightened vulnerability to adversarial attacks. This paper proposes a domain assimilation approach that introduces texture and color adaptation into transfer learning, followed by a texture preservation component to suppress undesired distortion. We systematically analyze the performance of transfer learning in the face of various adversarial attacks under different data modalities, with the overarching goal of fortifying the model’s robustness and security in medical imaging tasks. The results demonstrate high effectiveness in reducing attack efficacy, contributing toward more trustworthy transfer learning in biomedical applications. 
    more » « less
  3. null (Ed.)
    Denoising is a fundamental challenge in scientific imaging. Deep convolutional neural networks (CNNs) provide the current state of the art in denoising natural images, where they produce impressive results. However, their potential has been inadequately explored in the context of scientific imaging. Denoising CNNs are typically trained on real natural images artificially corrupted with simulated noise. In contrast, in scientific applications, noiseless ground-truth images are usually not available. To address this issue, we propose a simulation-based denoising (SBD) framework, in which CNNs are trained on simulated images. We test the framework on data obtained from transmission electron microscopy (TEM), an imaging technique with widespread applications in material science, biology, and medicine. SBD outperforms existing techniques by a wide margin on a simulated benchmark dataset, as well as on real data. We analyze the generalization capability of SBD, demonstrating that the trained networks are robust to variations of imaging parameters and of the underlying signal structure. Our results reveal that state-of-the-art architectures for denoising photographic images may not be well adapted to scientific-imaging data. For instance, substantially increasing their field-of-view dramatically improves their performance on TEM images acquired at low signal-to-noise ratios. We also demonstrate that standard performance metrics for photographs (such as PSNR and SSIM) may fail to produce scientifically meaningful evaluation. We propose several metrics to remedy this issue for the case of atomic resolution electron microscope images. In addition, we propose a technique, based on likelihood computations, to visualize the agreement between the structure of the denoised images and the observed data. Finally, we release a publicly available benchmark dataset of TEM images, containing 18,000 examples. 
    more » « less
  4. Recent advances show the wide-ranging applications of machine learning for solving multi-disciplinary problems in cancer cell growth detection, modeling cancer growths and treatments, etc. There is growing interests among the faculty and students at Clayton State University to study the applications of machine learning for medical imaging and propose new algorithms based on a recently funded NSF grant proposal in medical imaging, skin cancer detection, and associated smartphone apps and a web-based user-friendly diagnosis interface. We tested many available open-source ML algorithm-based software sets in Python as applied to medical image data processing, and modeling used to predict cancer growths and treatments. We study the use of ML concepts that promote efficient, accurate, secure computation over medical images, identifying and classifying cancer cells, and modeling the cancer cell growths. In this collaborative project with another university, we follow a holistic approach to data analysis leading to more efficient cancer detection based upon both cell analysis and image recognition. Here, we compare ML based software methods and analyze their detection accuracy. In addition, we acquire publicly available data of cancer cell image files and analyze using deep learning algorithms to detect benign and suspicious image samples. We apply the current pattern matching algorithms and study the available data with possible diagnosis of cancer types. 
    more » « less
  5. Abstract— Recent advances show the wide-ranging applications of machine learning for solving multi-disciplinary problems in cancer cell growth detection, modeling cancer growths and treatments, etc. There is growing interests among the faculty and students at Clayton State University to study the applications of machine learning for medical imaging and propose new algorithms based on a recently funded NSF grant proposal in medical imaging, skin cancer detection, and associated smartphone apps and a web-based user-friendly diagnosis interface. We tested many available open-source ML algorithm-based software sets in Python as applied to medical image data processing, and modeling used to predict cancer growths and treatments. We study the use of ML concepts that promote efficient, accurate, secure computation over medical images, identifying and classifying cancer cells, and modeling the cancer cell growths. In this collaborative project with another university, we follow a holistic approach to data analysis leading to more efficient cancer detection based upon both cell analysis and image recognition. Here, we compare ML based software methods and analyze their detection accuracy. In addition, we acquire publicly available data of cancer cell image files and analyze using deep learning algorithms to detect benign and suspicious image samples. We apply the current pattern matching algorithms and study the available data with possible diagnosis of cancer types. 
    more » « less