Age-related macular degeneration (AMD) is the leading cause of irreversible blindness in developed countries [20]. The number of people with AMD worldwide is projected to be 196 million in 2020, increasing substantially to 288 million in 2040 [22]. Based on clinical features, the disease is classied into early AMD, intermediate AMD (iAMD), and late AMD stages [12]. Late AMD is often associated with severe vision loss. Identifying patients at high risk of progression to late AMD, the sight-threatening stage, is critical for clinical actions, including medical interventions and timely monitoring.
Color fundus photography (CFP) is the most widespread and accessible retinal imaging modality; it is the most highly validated imaging modality for the detection of late AMD and the prediction of progression to the late stage of the disease [11]. Figure 1 show the CFP images of AMD progression for a patient's eye. Some followup visits (e.g., the third year's visit in Figure 1) might be missed. It takes 5 years for the eye to progress from early AMD stage to late AMD stage. Early identication of the risk of progression to late AMD and proper timely medical intervention might be able to alleviate disease progression.
Recently, deep learning methods have been proposed to detect abnormalities and AMD [4,5,7,14,18], and predict AMD progression [1,3,19,23] in coming years based on previous CFPs. Although the above methods have achieved superior performance, they are limited to CFPs in the last visit and do not include the longitudinal CFP history and AMD progression during the previous years' visits. The disease progression information can be found from the previous visits (as Figure 1 shows, patients usually have several visits before progression to late AMD). Patients in dierent subphenotypes might have various AMD progression speeds in earlier and later AMD stages. Capturing progression information during the last years' visits might be useful for the prediction of AMD progression.
In this study, we propose a novel late-AMD prediction framework ( Contrastive-Attention-based Time-aware Long Short-Term Memory network, CAT-LSTM) to model CFP sequences with consideration of AMD progression information. We utilize a time-aware long short-term memory (T-LSTM) to model patients' temporal visits with irregular time gaps. The input of T-LSTM consists of genotype information, sociodemographics, CFP feature vectors and AMD progression information. Following [19,23], we use the genetic risk score of 52 independent genetic markers as the genotype information, which has been reported to have associations with AMD risk in a recent large-scale genome-wide association study by the International AMD Genomics Consortium [13]. Following [3], we extract the smoking history, sex, age, race, body mass index as sociodemographic information. For the CFP features, we adopt DenseNet [16] to extract image feature vectors from CFPs. We introduce a contrastive attention module (CA) to remove the common features in the fundus image and learn fair image representation. We represent patients' AMD stages during the last years' visits as progression feature vectors. The concatenation of these four kinds of feature vectors is sent to T-LSTM to generate output vectors. Finally, fully connected layers and a Sigmoid function are followed to generate late AMD probability. Additionally, based on the learned xed-size representations of temporal CFP sequences, we adopt k-means to cluster them into various AMD subphenotypes.
To demonstrate the eectiveness of the proposed framework, we conduct experiments on publicly available a real-world dataset from Age-Related Eye Disease Study (AREDS) [21]. The experimental results show that the proposed models outperform the state-of-theart methods.
In sum, our contributions are as follows:
• The rest of the paper is organized as follows. In Section 2, we describe our model in detail. In Section 3, we conduct experiments on real-world CFP datasets AREDS. We review the related studies in Section 4. Section 5 concludes our work.
In this section, we propose a contrastive-attention-based timeaware LSTM (CAT-LSTM) to predict AMD progression and cluster CFP sequences into subphenotypes. We rst present a CNN model with a contrastive attention module (CA) to capture abnormality from CFPs, and time-aware LSTM (T-LSTM) to model the CFP sequences and predict late AMD risks in coming years. Then we cluster the learned CFP sequence representations into subphenotypes with k-means.
In this work, each patient's data consist of a sequence of visits, which include CFPs for both eyes and sociodemographics. The elapsed time between successive visits is irregular. We treat the two eyes of a same patient as independent samples. Given an individual eye of a patient, the CFPs are represented as
where ) denotes the number of visits for the patient. There are 9 steps (i.e., 1-9) for early and intermediate AMD stages, and 3 steps (i.e., [10][11][12] for late AMD stages [10]. The ground truth for the late AMD prediction tasks is represented as . = { ˆ1, ˆ2, ..., ˆ) }, where ˆC 2 {0, 1}. ˆC = 1 ( ˆC = 0) denotes the corresponding eye will (not) progress to late AMD stage in coming years. We set dierent prediction windows for the late AMD prediction tasks. The sociodemographics of the patient are represented as ⇡ = {3 1 , 3 2 , ..., 3 ) } 2 ' ) ⇥< , where < denotes the number of sociodemographic variables. This work aims to detect abnormalities, predict AMD progression from CFPs. The framework of the proposed CAT-LSTM is shown in Figure 2. Based on the features extracted by CAT-LSTM, we further study the AMD subphenotypes with a subtyping framework as Figure 3 shown. We list the important notations in Table 6.
Given a CFP E C in C C⌘ visit, we adopt DenseNet [16] to extract the image feature maps:
where 4 2 C 2 ' , ⇥ ⇥1024 is the output feature map before the average pooling layer of DenseNet. An attention module is adopted to automatically focus on the abnormal area in eye images.
where , V 2 ' 1024 , 1 V 2 ' are learnable parameters. We generate an AMD stage history vector based on previous visits and impute the missing AMD steps. Then we build a AMD progression time matrix to represent how many years it takes for the eye to progress from earlier AMD step to later AMD step. We atten the matrix and concatenate it with the imputed AMD step history vector to generate the progression embedding 4 ? C .
Contrastive attention module. Due to the imbalanced late AMD distribution on various patient groups (e.g., with dierent ages, gender, smoking history), deep learning models might learn the bias and discrimination from the data. We present a contrastive attention module to make model be fair and focus on abnormalities. We divide all the healthy eyes or early AMD stage eyes in training set into various pools based on demographics (i.e., gender, age, smoking history). Given an eye image of a patient, we rst collect the CFP feature vectors in the same pool as the patient, denoted as % = {? 1 , ? 2 , ..., ? |% | }, ? ⇤ 2 ' 1024 . An aggregate attention is introduced to generate the weighted average feature vector 4 2,? C as the common feature vector of the patient pool, where the weight is set as the cosine similarity. 4
To obtain the contrastive information 4 2,3 C , we remove (i.e., subtract) the common feature 4
There are 12 steps (i.e., step 1-12) and 3 categories (i.e., early AMD, iAMD, late AMD) for AMD disease [10]. To make the CNN model learn more accurate and ne-grained features, our model predicts the probabilities for the 12 AMD steps. Intuitive loss function for the multi-class classication task is cross-entropy loss. However, the loss function fails to consider the semantic gap between AMD steps (e.g., the dierence from step 4 to step 5 is much smaller than the dierence from step 4 to step 9), which might be harmful for the feature extraction and future late AMD stage prediction. Thus we convert the 12-class classication task to 12 binary-class classication tasks. Our model predict the ground truth ˆ2 C 2 ' 12 and ˆ2 C,8 (i.e., the 8 C⌘ dimension of ˆ2 C ) denotes whether the AMD step is higher than step 8 at time C. For example, given a CFP with AMD step equal to 3, we use ˆ2 C = [0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1] as the label vector. Given CFP feature 4 2,3 C , fully connected layers and Sigmoid layers are followed to generate the AMD probabilities for current visit:
where , ~,2 2 ' 1024⇥12 , 1 ~,2 2 ' 12 are learnable parameters. ~2 C 2 ' 12 denotes the probability for 12 AMD steps at time C.
where ~2 C,8 and ˆ2 C,8 are the predicted probability and ground truth for AMD step 8 at time C. After the AMD stage classication model is well trained, we assume the CNN model can extract the abnormal features from CFPs.
Sociodemographic embedding. Following [3], we extract patients' sex, age, race, body mass index and smoking history as sociodemographic information and represent them as binary vectors. Given the sociodemographic vector 3 C at C C⌘ step. We use fully connected layers to map 3 C to sociodemographic feature vector 4 3 C 2 ' : . Genotypic information embedding. Following [19,23], we incorporate 52 AMD-associated independent genetic variants reported by the International AMD Genomics Consortium [13] to the patient representation module. We map 52 AMD genetic risk score to vector 4 6 with fully connected layers. Progression information embedding. Following [3], we also utilize patients' previous AMD category information in late AMD prediction. Dierent from [3] that just concatenates the last visit's AMD category and demographic vectors, we present two kinds of progression information embedding methods to map previous AMD category sequences to embedding vectors.
In the rst method, we assume all the previous AMD categories have been correctly identied by clinicians. We represent the AMD progression information during the previous visits as a vector
consists of two kinds of information, as shown in Figure 2 (D). The rst is the AMD stage history during the last years' visits. We sample the AMD stage for every half year. Patients might miss some routine follow-ups and the AMD stage vectors are not fully observed. We impute the missing AMD stages with linear interpolation. The second kind of information of E ? C is the number of years that it takes for the individual eye to progress from step 8 to step 9, where 1 8 < 9 9. We concatenate the two vectors to generate E ? C 2 ' ; . In our implementation, we set the stage observation window as 6 years. Then we use a fully connected layer to map the progression vector E ? C to an embedding vector 4
where , ? 2 ' ;⇥: , 1 ? 2 ' : are learnable parameters. 4
? C denotes the AMD progression embedding, which contains the information how the AMD disease of the patient's eye progress during the last years' visits. We concatenate the multi-modal features 4 2,3 C , 4 3 C , 4
? C and 4 6 and adopt a fully connected layer to produce G C , which is sent to a time-aware LSTM to model the eye's health states.
where , G 2 ' 4:⇥: and 1 G 2 ' : are learnable variables. [•, •, •, •] denotes the concatenation operation.
In the second method, we use the stage prediction results ~2 C in Eq. ( 5) to generate progression embedding vector. The progression embedding method requires manual work for grading the AMD images. To develop an automatic late AMD prediction framework, we use the stage prediction results rather than the ground truth of previous AMD stages. Similarly, we utilize the predicted AMD categories during the last 6 years and conduct linear interpolation to impute the missing visits. Then we obtain a matrix . C 2 ' 12⇥12 to represent the previous AMD progression information. Then we adopt 1-D convolutional layers to map the vector sequences to produce progression vector 4 ? C 2 ' : .
To capture the temporal patterns of AMD disease progression, we utilize LSTM [15] to model the CFP feature sequences. To address the irregular time gaps between visits, following [2], we introduce time-aware LSTM, which adjust the memory vector ⇠ C 1 to ⇠ C 1 based on the time gap C as follows:
where , B 2 ' :⇥: and 1 B 2 ' : are learnable parameters. We rst divide the memory vector
and short-term memory ⇠ ( C 1 . The sum of long-term memory and discounted short-term memory is used as the adjusted memory vector ⇠ C 1 . 6( C ) serves as the discount function and we use 6( C ) = 4 C in our experiments. Given the adjust memory vector, we compute the new memory cell ⇠ C and hidden state ⌘ C :
where, 5 ,, 8 ,, > ,, 2 2 ' :⇥: , * 5 , * 8 , * > , * 2 2 ' :⇥: and 1 5 , 1 8 , 1 > , 1 2 2 ' : are learnable parameters. Fully connected layers and Sigmoid layers are followed to generate the late AMD probabilities:
where , ~2 ' : , 1 ~2 ' are learnable parameters. We use binary cross entropy to train the time-aware LSTM.
To further study the AMD progression patterns among dierent patients' eyes, we subtype individual eyes based on their temporal CFP sequences. We rst project the varying-length CFP sequences to vectors with CAT-LSTM.
As shown in Figure 3, the T-LSTM in Eq. ( 10) serves as the encoder to extract the temporal sequence information. We build another T-LSTM as the decoder to reconstruct the stage history of the previous CFP images. The hidden state and the cell memory of the T-LSTM encoder at the end of input sequence are used as the initial hidden state and the memory content of the T-LSTM decoder. The rst input time gap of the decoder is set to zero and it outputs the AMD stage in the last visit. Then the time gaps between visits are sent to T-LSTM decoder. When the reconstruction error is minimized, T-LSTM encoder learns the eective representations of CFPsequences.
where B C,8 and BC,8 are the predicted probability and ground truth for AMD step 8 at time C. Note that B C and BC have the same format as ~2 C and ˆ2 C in Eq. ( 6).
The auto-encoder and CAT-LSTM are jointly trained when subtyping CFP sequences. We use a hyper-parameter _(0 < _ < 1) to adjust the weights of the two loss functions.
; = _; ?AC + (1 _); 04 (14) Note that we only jointly train the late AMD prediction model and auto-encoder when subtyping CFP sequences. When conducting late AMD prediction experiments, we use the binary cross entropy loss in Eq. ( 12) to train CAT-LSTM.
To demonstrate the eectiveness of the proposed model, we conduct experiments on a real-world dataset and compare the proposed model with the-state-of-art methods.
The AREDS is a multi-center prospective cohort study of the clinical course, prognosis, and risk factors of AMD [21]. 4,612 participants aged 55-80 years are recruited from 1992 at 11 retinal specialty clinics in the United States. The inclusion criteria are wide, from no AMD in either eye to late AMD in one eye. The AREDS data set is publicly accessible to researchers by request at dbGAPfoot_0 . The statistics of the extracted data are displayed in Table 1. Note that in a visit, there might be multiple CFPs for individual eyes (e.g., from left and right sides). We randomly select one CFP in the training process, and use the average features from multiple images in the test phase.
We conduct experiments on two kinds of settings. The rst is late AMD detection: to detect whether the patients' eyes have progressed to late AMD stage. The second setting is late AMD prediction. Following [19,23], we predict whether patients' eyes will progress to late AMD within = years. Figure 4 shows the denitions of positive and negative samples. We conduct experiments with dierent n (i.e., n=1,2,3,4,5 and All). For patients who progress to late AMD at time C ; , the visits between time C ; = to C ; are set as positive samples, while the visits before time C ; = are set as negative samples. When n is All, we set all the visits before C ; as positive samples. For patients who do not progress to late AMD ultimately, C 4 is the time of last visit and all the visits before time C 4 = are set as negative samples. When n is All, we set all the visits before time C 4 as negative samples. The numbers of positive and negative samples in various settings can be found in supplementary materials (Table 9). Note that We remove the patients with less than 4 visits and the patients' eyes with late AMD in the rst visit when building the dataset.
We compare the proposed model with late AMD detection and prediction methods:
• DeepSeeNet [18]: DeepSeeNet is developed to automatically classify patients by the AREDS Simplied Severity Scale (score 0-5) using bilateral CFP. DeepSeeNet also has an output branch to generate the risk of late AMD stage. We modify the output branch to predict AMD progression within coming years. • Chen et al.
We implement our proposed CAT-LSTM models with PyTorch 0.4.1 3 .
For training models, we use Adam optimizer with a mini-batch of 8 patients. For each patient, we send the CFP images in recent 8 visits to our model. We train on 4 GPUs (TITAN RTX 2080) for 40 epochs, with a learning rate of 0.0001. We randomly divide the patients in the dataset into 10 sets. All the experiment results are averaged from 10-fold cross validation, in which 7 sets are used for training 3 https://pytorch.org/ every time, 1 set for validation and 2 sets for test. The validation sets are used to determine the best values of parameters in the training iterations. We rst pretrain CA-CNN with loss ; 2;B in Eq. ( 6) and then train the whole CAT-LSTM model with loss ; ?A3 in Eq. ( 12). We use the area under the receiver operating characteristic curve (AUROC) in the test sets as a measure for comparing the performance of all the methods. The CFP images, sociodemographics, genotype and progression information are projected into 1024-d vectors. We set the hidden state of T-LSTM as 256-d vectors. More details can be found at GitHub 1 .
Table 2 displays the late AMD detection and prediction results. The results show that all the methods achieved similar performance on the late AMD detection task. The reason is that when an eye progress to late AMD stage, the eye images would look very different from healthy eyes (e.g., large drusen area or pigmentary abnormalities), as shown in Figure 1. Deep learning models can easily capture the abnormalities and accurately classify the images. In this subsection, we mainly discuss the late AMD prediction tasks, which are more signicant for timely clinical intervention for high-risk patients. The proposed model signicantly outperforms the baselines on the late AMD prediction tasks on AUROC (all % E0;D4B < 10 5 ), which demonstrates the eectiveness of our model. Among the baseline methods, with the genotype information, Yan et al. [23] and Peng et al. [19] outperform the other models, the reason is that the two methods incorporate more information from the genetic risk scores of 52 independent genetic markers. Babenko et al. [1] performs better than DeepSeeNet [18], Chen et al. [7] and Bhuiyan et al. [3], we speculate the reason is that the model takes both left and right sides of same eyes as input and thus can capture the global features of the eyes. Both versions of our model in Table 2 outperform the baselines, which can demonstrate the eectiveness of our model. With additional ground truth information of AMD stage in previous visits, CAT-LSTM-v1 outperforms the baselines by more than 3% on AUROC. To fairly compare with the baselines, CAT-LSTM-v2 takes the predicted AMD stage by our model as inputs, and still outperforms the baselines. Table 2 shows that each model achieves similar superior performance in dierent settings of late AMD prediction, while shortterm prediction is supposed to be easier than long-term prediction. We speculate the reason might be related to imbalanced positive/negative distribution in dierent settings. As Table 9 in supplementary materials shows, the numbers of positive samples in short-term late AMD prediction settings (e.g., < 2% positive samples in 1-year late AMD prediction setting) are much less than long-term AMD prediction settings (e.g., > 15% positive samples in 5-year late AMD prediction setting), so the short-term prediction models cannot outperform the long-term prediction models with much less positive samples. Our prediction results also align with existing late AMD prediction studies [19,23].
Ablation study. To further investigate the contribution of each component (i.e., contrastive attention module, AMD progression embedding, genotypic information embedding) of our model, we conduct ablation study by comparing additional four versions of the proposed model:
• CNN: We only use a CNN model (i.e., DenseNet) to predict the risk of progression to late AMD stage after 5 years. • CA-CNN: To demonstrate the eectiveness of the contrastive attention module, we use the CNN model with contrastive attention module to predict the risk of progression to late AMD stage after 5 years. • CA-CNN+Genotype: To demonstrate the eectiveness of genotype information, we incorporate the genotype embedding vector when representing patient health states and predict the risk of progression to late AMD stage after 5 years. • CAT-LSTM ? : To demonstrate the eectiveness of progression information, we remove the progression embedding and only use CFP images, genotype and sociodemographics information to predict AMD progression. Since area under Precision-Recall curves (AUPRC) give a more informative picture of an algorithm's performance than AUROC in imbalanced datasets [9], we add AUPRC to evaluate the various versions of our model. Table 3 displays the prediction results on AUROC and AUPRC. CA-CNN outperforms CNN by more than 2% on AUPRC, which demonstrates the eectiveness of contrastive attention module. We speculate the reasons are two-fold: (i) contrastive attention module force the model to focus on the dierence between the input image and healthy eye images, which might be the abnormal regions of the input image; (ii) the late AMD stage rates among dierent patient groups vary a lot as shown in Table 8, contrastive attention module remove the common information in Progression time(S1-S4) 5.6 years 2.8 years 1.9 years Progression time(S4-S9) 4.9 years 4.7 years 3.6 years patient groups, which make the feature representation fairer and thus can improve the overall prediction performance. With the consideration of genotypic information, CA-CNN+Genotype outperforms CA-CNN on both AUROC and AUPRC, which demonstrates that genotype data could improve the AMD progression prediction performance. CAT-LSTM ? outperforms the CA-CNN+Genotype, which demonstrates that with previous visits' information, T-LSTM can improve the prediction results by capturing the AMD stage progression information. Inspired by the thought, we directly embed the progression information and input to our model in CAT-LSTM-v1 and CAT-LSTM-v2. With the ground truth of AMD stages in previous visits, CAT-LSTM-v1 outperform the other versions a lot on both AUROC and AUPRC. Even without manually labeled AMD stage as inputs, CAT-LSTM-v2 still performs much better than other versions, which further demonstrates capturing previous AMD progression information does help the prediction for future AMD progression.
We cluster the patients' eyes based on their fundus image sequences. The auto-encoder can output the previous AMD stages during the last years' visits. We can assume the hidden state vector input to the decoder contains the progression information and can represent the whole fundus image sequence. We cluster the hidden state vector with k-means and obtain 3 subphenotypes. Note that we only use patients' rst 4 years' visits of early and intermediate AMD stages for clustering. Subphenotype characteristics. The cluster descriptive statistics of AMD subphenotypes are shown in Table 4. The subphenotype I has the most individual eyes, youngest age and the lowest AMD stage rate. The subphenotype III has the least individual eyes, but more than half the eyes in the subphenotype would progress to late AMD stages. We nd that age and smoking history have positive correlations with the late AMD stage rate. Moreover, subphenotype III has the maximal abnormal rates (e.g., depigmentation, Existing study [13] has reported that 52 independent genetic variants are associated with AMD disease. We compute the distribution of the genetic variants across 3 subphenotypes. Figure 7 displays the genetic variants with signicant distribution dierence (P-value < 0.05) across the 3 subphenotypes. Because the alternative allele rates of dierent genetic markers vary a lot, we normalize the rates when visualizing the distribution of the genetic variants. Based on the dierent characteristics (e.g, sociodemographics, abnormalities, AMD progression speed in Table 4, and genetic variant distribution in Figure 7), we might be able to early identify patients' AMD subphenotype and provide timely medical interventions for patients with high risk of progression to late AMD stage, which might improve the treatment eects or alleviate the disease progression.
Clustering evaluation. To evaluate the clustering performance, we compare the proposed CAT-LSTM with a baseline CNN+LSTM, which adopts CNN to represent CFP as feature vectors and leverage LSTM to extract CFP sequence features. We use the same k-means method to group the CFP sequences for CNN+LSTM. Table 5 displays the clustering results. Since we do not know the ground truth AMD subphenotypes, we cannot measure the clustering performance with common clustering evaluation metrics such as purity and rand index. Instead, we use two popular metrics Calinski-Harabasz Index (CHI) [6] and Davis-Bouldin Index (DBI) [8], which can measure the performance of clustering algorithms on labelunknown dataset. Note that CHI is related to the size of the dataset, we normalize the value by dividing CHI by the number of patients. The results show that both versions of CAT-LSTM perform better than CNN+LSTM on both metrics. Visualization of learned CFP sequence representations. We adopt the t-distributed stochastic neighbor embedding (t-SNE) algorithm [17] to project all CFP sequences of individual eyes into a 2D space and Figure 5 shows the visualization results. To fairly compare with CNN+LSTM, we use CAT-LSTM-v2 (which embeds progression features based on predicted AMD stage rather than ground truth) to extract CFP sequence features. Based on which stage the individual eye progress to ultimately, we divide the eyes into 3 groups (i.e., early AMD stage, iAMD stage and late AMD stage) with dierent colors in Figure 5. Note that we just use the CFP sequences up to iAMD stage for Figure 5
In this section, we briey review the existing works related to late AMD detection and prediction.
AMD Detection. Burlina et al. [5] compare the performance of humans and deep learning in grading CFP to detect AMD. Peng et al. [18] present DeepSeeNet to classify patients automatically by the AREDS Simplied Severity Scale (score 0-5) using bilateral CFP. Grassmann et al. [14] utilize an ensemble of neural network architectures to classify CFPs into 12-step categories. Burlina et al. [4] adopt deep convolutional neural networks to predict the AREDS 9-step detailed severity scale for AMD to estimate 5-year risk probability with reasonable accuracy. Chen et al. [7] detect four AMD characteristics (drusen area, geographic atrophy, increased pigment, and depigmentation), then combine them to derive the overall 9-step score.
AMD prediction. Babenko et al. [1] adopt Inception-v3 to predict AMD progression for stereo pairs of dierent sides of the same eye. Yan et al. [23] utilize Inception-v3 to extract deep image features. The deep features with 52 independent genetic variants are fed to another fully connected layer to predict the time to late AMD development exceeding certain inquired years. Peng et al. [19] combine the deep features generated by DeepSeeNet and genotypic information to represent the patients' eyes, then adopt a survival model to predict AMD progression risk. Bhuiyan et al. [3] propose to ensemble ve deep learning frameworks (e.g., Inception-V3, Xception, Inception-Resnet-v2) to predict AMD progression. The resulting AMD scores of various models are combined with sociodemographic clinical data (including age, race, sex, body mass index, visual acuity, and sunlight exposure) and other automatically extracted imaging data by a logistic model tree machine learning technique to predict risks of progressing to late AMD.
Although the methods described above have achieved superior performance on AMD detection and prediction, they do not make full use of the CFP sequence data and AMD progression information in late AMD prediction tasks, which might limit their prediction performance. In this study, we present progression embedding and introduce T-LSTM to capture AMD progression information, which signicantly improves the prediction performance.
We proposed a new AMD progression prediction framework CAT-LSTM. The framework adopts CNN to extract fundus image features and a time-aware LSTM to model CFP sequence feature, sociodemographics, genotype and AMD progression information. We introduce a contrastive attention module to force the framework to focus on the abnormal area of images. To explicitly utilize the AMD progression information during the last years' visits, we present a progression embedding module to map the AMD step sequences to a vector. Experiments on real-world datasets have shown that all the contrastive attention modules, progression embedding, and T-LSTM can improve late AMD progression performance. Moreover, we represent patients' temporal image sequences as xed-size vectors with an auto-encoder and subtype the CFP sequences with k-means based on the learned representation. The subtyping analysis shows that the patients in the 3 subphenotypes have dierent probabilities of progressing to the late AMD stage. The proposed AMD subtyping framework is useful in identifying patients with a high risk of progressing to the late AMD stage in patients' iAMD stage, which paves the way for improved personalization of AMD management.
We list the important notations in Table 6.
)
The number of visit +
The CFP sequence 6
The genotype vector 4 6 The embedding vector of 6 E C
The CFP image in the C C⌘ visit 4 2,0
The CFP feature generated by attention module ? 8
The CFP feature vector in patient pool 4
There is a hyper-parameter _ in Eq. ( 14). Note that we only jointly train the late AMD prediction model and auto-encoder when subtyping CFP sequences. We use the clustering evaluation metrics CHI and DBI to select the value of _. As Table 7 shown, when 0.3 _ 0.9, clustering performance is not sensitive to _. In our experiment, we set _ = 0.5. We try to use dierent K for k-means when clustering the CFP sequences. As shown in Figure 6, when = 3, we have the best DBI value for CFP clustering. It is also the elbow point for CHI. Thus we cluster the CFP sequences into 3 subphenotypes.
Patients with dierent demographics (e.g., gender and age) have dierent risks of progressing to late AMD stages.
We further analyze the alternative allele distribution of 52 AMDassociated genetic markers across various subphenotypes. Figure 7 displays the alternative allele distribution of AMD-associated
https://www.ncbi.nlm.nih.gov/projects/gap/cgi-bin/study.cgi?study_id=phs000001. v3.p1