Chest X-ray (CRX) is a vital tool for quick patient triage and as such, there have been great efforts to make computer analysis of X-ray images possible. As imaging technology advances, the number of images per patient continues to grow. Reliable and fast automated analysis can alleviate the workloads for practitioners by offloading some of the work to a computer. However, there are three key challenges in automating the analysis of CRX images.
First, medical images come in various modalities including computed tomography (CT) and traditional X-ray. CXRs may be captured at different angles determined by the patient's status and the physician. Consequently, the analysis model needs to detect patterns varying across these different modalities and learn the relationships between them. The multi-image nature of the data can result in a large number of features that requires significant computational resources to process. Second, multiple images can be captured across different points in time. At the same time, some images at specific times are captured while others are not. For example, X-rays are more accessible and cost-effective than CT which can lead to some angles having X-rays but no CT. Finally, the images can be collected from different devices and hospitals, resulting in discrepancies in the image format and resolution. Existing machine learning models assume a fixed-size image, so they rely on rescaling methods which may incur a loss of information (from down-scaling) or undesirable bias (from upscaling). Lost information resulting from rescaling negatively impacts a model's performance.
The previous research has framed image analysis as a multiinstance learning problem (MIL) for two reasons: the number of images per patient differ across the dataset and individual images may not be labeled. MIL [1], [2], [3] is a weaklysupervised learning model, which is ideal for this application as each patient is in the form of a labeled "bag". Labels are associated with the bag, not the individual images, so the clinician does not need to label each image individually. There have been extensive studies into machine learning algorithms for MIL including support vector machines (SVMs) and deep learning models. Some examples of SVMs are Multi-Instance Support Vector Machine (MISVM) [4], sparse Multi-Instance Learning (sMIL), sparse balanced MIL (sbMIL) [5], Normalized Set Kernel (NSK), and Statistics Kernel (STK) [6]. These methods have successfully labeled the bags in the testing dataset as either malignant or benign. The multi-instance deep learning models include mi-Net and MI-Net [7], and more recently the attention mechanism-based models such as Attention-based deep Multiple Instance Learning (AMIL) [8] and Loss-based Attention Multiple Instance Learning (LAMIL) [9] are gaining popularity.
Although these models already exceed human performance in some applications, the success of those models depends on extensive training time and computational resources. Considering these difficulties, based on our earlier works [10], [11], [12] in this paper we propose a Fast Multi-modal Multiinstance Support Vector Machine (FMMSVM) method to improve the performance and effectiveness of CXR analysis. Our contributions can be summarized into the following:
• The proposed model simultaneously imputes the missing modality and predicts the clinical outcomes in spite of missing data. This joint imputation is designed to estimate the values of missing entries most helpful for predicting diagnoses. • We derive an efficient solution algorithm for the proposed FMMSVM which linearly scales to the number of features of input data reducing the need for training time and computing resources.
Throughout the remainder of this paper, we denote matrices with bold upper-case letters (e.g., M), vectors as bold lowercase letters (e.g., m), and scalars as lower-case letters (e.g., m). The i-th row and j-th column of matrix M are written as m i or [M] i and m j or [M] j . The scalar value indexed by the i-th row and j-th column of M are written as m i j or [M] i j . Each i-th bag
We denote the trace norm of a matrix as tr[•].
We start our formulation with the K-class multi-instance SVM [4]:
(1) Here N denotes the total number of bags, representing patients, the hyperplane w y and bias b y are associated with the positive class label for i-th bag X i . However, the conventional multi-instance SVM in Eq. ( 1) is not able to reach ideal performance with the missing data. To overcome this limitation, we are motivated to develop FMMSVM to jointly perform the clinical outcome prediction and imputation as:
where d j,k i is time interval (i.e., temporal distance) between jth and k-th instance. The imputation loss L imputation contains factorization [13], [14], locality preserving projection [15], and trace norm [16] terms. The mask M i ∈ d×ni contains the binary missingness information of data X i , where 1 indicates the known entry and 0 indicates the unknown/missing entry. Additionally, is Hadamard product. F ∈ d×ni is the imputed matrix of X i which keeps the known entries in X i . The trace norm is defined as
], which improves the smoothness between the imputed and known entries.
Both factorization and trace norm terms discover low rank structure of input data X i and unknown entries are extrapolated using with linear combination of known entries. This is accomplished by minimizing factorization term
, where the known entry in X i is expressed by the product between row of H ∈ d×r and column of Z i ∈ r×ni , and unknown entries are imputed by H and Z i learned from the known entries. The dimensionality r of Z i is typically much less than d of X i . Therefore, Z i represents the enriched version of X i which removes the redundant information in X i , and we replace X i in Eq. ( 1) with our learned representation Z i . As a result, the decision function is given as such ỹi = arg max m (max(W T H + X i + b1 i ) m ) , where H + is Moore-Penrose pseudo-inverse of H.
From the imputation integrated SVM in Eq. ( 2), we should consider the two important aspects in the images learning. First, for each patient the multiple images are captured across the different time points. Therefore, disease patterns in the two consecutive images captured at the similar time points tends to be associated each other. Second, the medical images are provided in the multiple modalities (e.g., X-ray or CT) and some modalities can be more predictive than the others. To account for these two factors, we introduce the graph learning to preserve the temporal locality where the inverse of temporal distances each pair of instances (nodes) are weights (similarities). We also factorize F modality by modality to learn H shared across all the bags and Z i shared across all the modalities. The hyperparameters α g adjust the importance of each modality.
Although the factorization and regularizations we have introduced in Eq. ( 2) are highly motivated, it adds many terms and complexity to our objective. Following our previous studies [11], we split the primal variables in Eq. ( 2) via Alternating Direction Method of Multipliers (ADMM) [17] approach. Another difficulty in the derivation is that the regularization terms in Eq. ( 2) are non-smooth and the gradients may not exist at some points. To improve the stability of the optimization, we use the optimization framework of the earlier work [18] that propose the iterative reweighted method to minimize non-smooth objective in Eq. ( 2) in which the key step is minimizing the following smoothed objective: min W,b,F,H,Z,B,E,Q,R,T,U τ
where D 0 ∈ d×d is a diagonal matrix whose j-th diagonal element is α g when j-th feature of instance is in g-th features group (modality).
and S is a diagonal matrix where each diagonal element is the row (or column) sum of S i such that
∈ D×D and δ is a small constant value for smoothness.
From Eq. ( 3) we derive the following equation using the augmented Lagrangian method: b,F,H,Z,B,E,Q,R,T where W,b,F,H,Z,B,E,Q,T,R, U are the primal variables, Λ 1,i , Λ 2 , Λ 3,i , Λ 4,i , Σ, Θ, Ω, Ξ, Γ are the dual variables.
The algorithm to solve the above objective is provided in Algorithm 1. The detailed derivations of the algorithm is not provided here due to space limit and will be provided in the extended journal version of this paper.
The chest X-rays are commonly utilized in medical research and clinical practice to detect abnormalities. In our experiments, we use an publicly available dataset of chest X-ray and CT images. Each patient (bag) is labeled by in-hospital mortality and associated with multiple images recorded across the different time points. For each time step we have two CXRs captured from the front and side as well as one CT image. The images are collected from the different public sources, hospitals, and physicians, which results in different shapes between images. The Fig. 1 and2 shows the widths, heights, and ratios of images in the dataset, and we observe the high variance in the size and ratio distributions. As a result, rescaling the images with the interpolation method can significantly distort the objects in the images. Therefore, instead of rescaling, we divide each image into 3×4 patches. Each patch is then vectorized through Parameter Free Threshold statistics Algorithm 1 The multiblock ADMM updates to optimize Eq. ( 4) b, F, H, Z, B, E, Q, R, T, U anddual variables Λ 1 , Λ 2 , Λ 3 , Λ 4 , Σ, Θ, Ω, Ξ, Γ. 4: while residual > tolerance do 5:
Update D 1,i (i ∈ {1, • • • , N}), D 2 by Eq. ( 3). 6:
for m ∈ K do 7:
Update wm ∈ W by Eq. (??). 8:
Update bm ∈ b by bm = 9:
end for 11:
for (i, m) ∈ {N, K} do 12:
Update e m p ∈ E by
13:
where
Update q m i ∈ Q by 16:
Update r m i ∈ R by 18:
for j ∈ n i do 20:
Update t m i,j ∈ T by 21:
Update u m i,j ∈ U by 24:
Update F i by Eq. (??).
Update Λ 1,i , Λ 3,i , Λ 4,i by 37:
end for 42:
Update H by (PFTAS) [19] method. These patches from two CXRs and one CT image at each time point is concatenated to create each information-dense instance. The PFTAS extracts the texture features by counting the number of neighboring black pixels for each pixel. Then the total count for all the pixels in a given patch is stored in a nine-bin histogram [19]. The thresholding is conducted by Otsu's algorithm [20] which generates a 162dimensional feature vector for each patch. As a result, each instance is a vector of 162 × 12 × 3 features. We list the details of this dataset:
• Cohort size: The number of patients is 323. The number of all patients associated with this dataset is 472, however 149 patients have been dropped because their labels are not provided.
We compare the classification performance and scalability of proposed FMMSVM to the following models:
• (1) A single-instance learning (SIL) method that assigns the bags' labels to all instances during training and produces the maximum response for each bag/class pair at testing time for the training bag's instances.
For the classification models used in Table I, we report the following hyperparameters found by grid search on the balanced accuracies of five test sets. For SIL, NSK, and STK the regularization tradeoff is set to 1.0. We set τ 0 , τ 1 , τ 2 , τ 3 , initial μ to 1e + 2, 1e -1, 1e -2, 1e + 2, 1e -5 for our exact FMMSVM model and 1e+2, 1e-2, 1e-2, 1e+3, 1e-10 for our inexact FMMSVM model. The tolerance is set to 1e -5 for both. We set α g to 5.0, 3.0, and 1.0 for front-view CXR, side-view CXR, and CT image modality. The deep learning models (mi-Net, MI-Net, MMMI-deep, AMIL, and LAMIL) are implemented using the codes provided by their respective papers [7], [21], [8], [9].
In table I, we report precision, recall, F1-score, accuracy, and balanced accuracy (BACC) in classification of survival/death bags. We split the bags into 80% for training and 20% for test set, then we train the classifier with training set of 3 folds, and tune the hyperparameters based on the accuracy on the validation set of 1 fold. Finally, the performance is measured on the test (held out) set of 1 fold and this is repeated 5 times and scores are averaged across 5 results following 5-fold cross validation scheme. The comparison between the classification models in Table I shows that the proposed FMMSVM models outperform the other existing multi-instance models. We interpret that our model has the better capability in joint imputation-prediction and learning temporal relationships between the instances. When Ours is compared to Ours w/o LP and Ours w/o SL, we observe that the introduced locality preserving and smoothness loss terms improve the prediction. These results demonstrate that classification pattern for FMMSVM can vary based on the optimization strategy.
Although the results from CXR dataset are promising, the cohort size (323 patients) is limited and the classification score gaps between our model and best performing baseline model (MMMI-deep) in Table I is within 3σ. Therefore we have extended our experiments to the additional dataset, which can provide the better measurement on the clinical applicability. We have conducted experiments on Alzheimer's Disease Neuroimaging Initiative (ADNI) database collected from 818 participants. In this dataset, each participant (bag) has 1 to 5 brain MRIs (1 to 5 instances), and we perform FreeSurfer and voxel-based morphometry [22] to extract the gray matter measures for 90 regions of interest for each brain MRI. To reproduce the missing modalities, we discard each MRI randomly with 50% probability. We classify each patient's Alzheimer's Diesase progression (AD) in {Alzheimer's Disease (337 AD patients), Mild Cognitive Impairment (251 MCI patients), Healthy Control (230 HC patients)}. Considering this ternary classification is usually more difficult than binary classification, the results are promising and they show the significantly improved performance of our model as the score gaps are larger than 3σ.
Besides the improved performance, our model has identified the AD risk factor as shown in Fig. 3. We have analyzed the learned weights of our model on each ROI feature. Our model identifies hippocampus [23], caudate nucleus [23], lateral ventricle [24], and amygdala [25] regions. The identified regions have been shown in the medical literatures to be related to AD. These results additionally validate the correctness of disease progression prediction from our model.
Information in medical image datasets usually represent a large number of features and are delivered in a variety of modalities. As data mining technologies develop, multimodal methods are attracting more attention in the field of machine learning researches. In this study, we present a novel multi-instance learning model that is scalable to a large number of features. We employ the factorization based joint imputation-prediction approach to handle the missing data in the modalities and PFTAS methods to control the fine-grained details of imaging information. In our experiments, we have observed promising performance and scalability of the proposed method when compared to the existing SVM and deep learning models. In addition to the improved performance and scalability, our model identifies the disease relevant regions in the images.