Retinal imaging, particularly fundus photography, is a noninvasive technique widely used in ophthalmology to capture detailed visualizations of retinal structures. By analyzing these images, clinicians can diagnose not only ocular diseases but also systemic conditions such as hypertension and diabetes (Li et al. 2023;Tan et al. 2024). However, manual interpretation by ophthalmologists is costly, timeconsuming, and subject to variability, potentially leading to delays in patient care and inconsistent diagnoses. Therefore, there is an urgent need for automated tools to improve disease detection efficiency through retinal image analysis.
Deep learning has emerged as a promising tool for automating disease detection using retinal images (Zhou et al. 2023;Huang et al. 2023;Zhao et al. 2023). These methods typically leverage established computer vision architectures and employ transfer learning to adapt them for various medical applications, as illustrated in Figure 1(a). For example, RETFound (Zhou et al. 2023), built on the Vision Transformer (ViT) architecture, is pretrained on large-scale datasets and later fine-tuned on retinal image datasets for disease detection. However, despite their effectiveness, these approaches focus primarily on modeling the overall data distribution of retinal images rather than on highlighting structural features of the retina. Critical diagnostic features are often embedded in the fine-grained structural details of the retina elements that may not significantly impact the overall data distribution but are essential for accurate disease diagnosis. Consequently, compared to natural image classification tasks, retinal disease diagnosis requires models with a stronger ability to capture and interpret key structural features. To address this challenge, a common strategy is to segment key retinal structures from retinal images (Li and Liu 2022;Almeida et al. 2024;Wang et al. 2021a;Sivapriya et al. 2024). By isolating diagnostically significant structures, the model can focus on extracting relevant features, as shown in Figure 1(b). For example, (Almeida et al. 2024) utilizes a customized image processing technique to segment retinal blood vessels and feed them into DenseNet121 for disease classification, while (Sivapriya et al. 2024) employs ResEAD2Net for blood vessel segmentation and sub-AP AP 3×3 Conv Sigmoid SGA Segmentation Module Feature Map SGA Segmentation Guided Feature Map Segmentation-Guided Attention Block C Concatenation Element-wise Multiplication AP Average Pooling Raw Image Feature Map Segmented Image Encoder Segmentation Module Raw Image Encoder … C MLP MLP Multilayer Perceptron SGA SGA SGA SGA sequently applies multiple machine learning algorithms to the segmented data for disease prediction. Although these methods improve attention to segmented regions, they ignore valuable information from complementary image areas, potentially limiting overall diagnostic performance.
To extract more comprehensive features, recent studies have integrated both segmentation results and retinal images for disease diagnoses (Alam et al. 2023;Joshi, Sharma, and Dutta 2024;Xiong et al. 2025). Specifically, some approaches fuse segmented and raw images into a single input and then feed it into an encoder for classification, as shown in Figure 1(c), while others process segmented and raw images through separate encoders to extract features for disease classification, as shown in Figure 1(d). For example, (Alam et al. 2023) stacks segmentation maps and retinal images into a single input for GoogleNet, whereas VisionDeep-AI (Joshi, Sharma, and Dutta 2024;Xiong et al. 2025) concatenates features extracted from segmented images and retinal images using separate EfficientNet or ResNet50 models. However, these methods lack explicit interactions between segmentation and classification feature spaces. As a result, retinal anatomical features are not fully leveraged to enhance the learned representations in the classification model, limiting the model's ability to incorporate prior structural information for improved disease diagnosis.
In this paper, we propose SegImgNet, a deep learning framework for retinal disease classification that integrates both retinal images and segmentation maps. By leveraging multi-scale structural feature maps obtained from segmentation along with original retinal images, SegImgNet enhances classification performance. The framework consists of two main components: a segmentation module and a classification module. The segmentation module, based on the U-Net (Ronneberger, Fischer, and Brox 2015) architecture, generates retinal structure feature maps. The classification module includes a segmented image encoder, a raw image encoder, a classifier, and Segmentation-Guided Attention (SGA) blocks. The segmented image encoder extracts disease-related local features, while the raw image encoder captures broader global contextual information. Both encoders are built on the ConvNeXt architecture, and the classifier combines their outputs into a unified representation for disease classification. Additionally, the SGA block enhances feature extraction by generating attention maps from structural segmentation, allowing the model to focus on critical retinal details. Extensive experiments on public AIROGS and private e-ROP datasets demonstrate that SegImgNet consistently outperforms existing state-of-theart methods for retinal disease diagnosis.
Figure 2 illustrates the architecture of SegImgNet, which consists of two main components: a segmentation module and a classification module. The details of these two modules are introduced below.
The segmentation module f seg (•) employs a U-Net architecture to generate retinal structure feature maps. U-Net utilizes a symmetric encoder-decoder architecture with skip connections, enabling it to capture both low-level spatial details and high-level abstract features. This structure ensures the precise localization of the retinal structures while preserving fine-grained anatomical details.
The U-Net's encoder consists of multiple convolutional layers followed by downsampling operations, progressively reducing spatial resolution while enhancing feature abstraction. This hierarchical representation enables the model to capture retinal structures across multiple scales, which is essential for detecting both fine-grained details and broader pathological patterns. The decoder, on the other hand, reconstructs the segmented image by gradually upsampling the encoded features, restoring spatial details lost during downsampling. Skip connections bridge the corresponding encoder and decoder layers, allowing high-resolution features from the encoder to be directly merged with upsampled features in the decoder. These connections help preserve fine-grained structural information, which is crucial for accurately delineating retinal regions.
Specifically, given a retinal image x ∈ R H×W ×Craw , where H, W , and C raw denote the height, width, and channel size of the raw image, respectively, the corresponding segmented image x seg ∈ R H×W ×Cseg and multi-scale retinal structural feature maps {h
2 i ×Ci are obtained as follows:
where C seg represents the channel size of the segmented image, and L represents the number of feature scales, which is empirically set to 4 in this study (Li et al. 2024).
The classification module extracts structural features from segmentation maps and contextual representations from raw retinal images for disease diagnoses. It consists of a segmented image encoder, a raw image encoder, a classifier, and SGA blocks. Each component is detailed below. Segmented Image Encoder: The segmented image encoder extracts fine-grained structural representations from the output of the segmentation module while incorporating segmentation priors at multiple stages. Here we use ConvNeXt (Liu et al. 2022) as a feature extractor or backbone for this encoder. Each stage of the feature extractor is equipped with a Segmentation-Guided Attention (SGA) block, which enhances attention to retinal structural features. By selectively emphasizing relevant features and filtering out less informative regions, the SGA block ensures that the extracted representations retain critical anatomical details essential for accurate disease classification and improved diagnostic reliability. The final segmentation map feature representations are obtained from the last stage of the feature extractor, where segmentation-guided information is further enriched with anatomical details. Specifically, the SGA block builds on the approach in (Li et al. 2024), utilizing convolution operations and a sigmoid activation function to refine feature extraction. It enhances the intermediate feature maps of the segmented image encoder by integrating segmentation-derived structural information. Given the output feature map h (i) local from the i-th stage of the feature extractor and the corresponding retinal structural feature map h (i)
seg , the SGA block produces an enhanced representation, formulated as:
where Conv 3×3 (•) represents a convolutional layer with a kernel size of 3 × 3 used for adjusting h
seg spatial dimensions to match h Raw Image Encoder: The raw image encoder is designed to extract global contextual representations from retinal images, complementing the structural features extracted by the segmented image encoder. Similarly to the segmented image encoder, it employs ConvNeXt as the backbone. However, unlike the segmented image encoder, which processes segmented images with segmentation-derived feature map enhancement, the raw image encoder focuses on capturing broader disease-relevant patterns within the retinal image. In particular, the raw image encoder is not equipped with SGA blocks, ensuring that it does not emphasize the same structural features as the segmented image encoder. This design preserves feature complementarity by allowing the segmented image encoder to prioritize segmentation-guided structural information. The final global feature representations are obtained from the deepest stage of ConvNeXt, where high-level disease-relevant information is encoded while retaining spatial context. Classifier: After obtaining the segmented image feature embedding h local and the raw image feature embedding h global from the encoders, the classifier concatenates them to form a comprehensive feature embedding h cls for disease classification. It then applies a Multilayer Perceptron (MLP) followed by a sof tmax activation function to classify diseases based on the feature embedding h cls . Specifically, the probability of the k-th disease, ŷk , is computed as follows:
where K denotes the total number of classes, and f k (h cls )) represents the MLP output for class k.
To address the class imbalance commonly found in medical datasets, we employ a Weighted Cross-Entropy (WCE) loss function to train SegImgNet. This loss function assigns higher penalties to misclassified minority-class samples, mitigating the dominance of majority classes and improving the model's ability to detect rare disease cases. The WCE loss is defined as:
) where N is the number of input samples, w k denotes the weight assigned to class k. y
Datasets: We evaluated SegImgNet on two datasets: the public AIROGS dataset and the private e-ROP dataset. • The AIROGS dataset (Steen et al. 2023) is an improved glaucoma dataset consisting of a balanced subset of standardized retinal images. It is derived from the Rotterdam EyePACS AIROGS set, which contains 113,893 color retinal images from 60,357 subjects across approximately 500 different sites with heterogeneous ethnicities. These retinal images were labeled as glaucomatous or healthy based on clinical evaluations performed by glaucoma specialists. For this study, we used 4,950 publicly available retinal images, including 2,475 glaucomatous images and 2,475 healthy images.
• The e-ROP dataset originates from the Telemedicine Methods for Evaluating Acute Retinopathy of Prematu-rity (e-ROP) study (Quinn et al. 2014), which collected retinal images from 1,257 infants admitted to neonatal intensive care units across 13 centers in North America. These images are captured using wide-angle retinal cameras during scheduled diagnostic examinations. Each retinal image was labeled as either preandplus or normal by experienced ophthalmologists. In this study, we used 7,811 center-view retinal images, including 990 preandplus images and 6,821 normal images. Evaluation Metrics: We evaluated model performance us-ing six standard metrics: Area Under the Receiver Operating Characteristic Curve (AUC) to assess discriminative ability, sensitivity (true positive rate) to quantify disease detection capability, specificity (true negative rate) to measure the ability to identify healthy cases, precision (positive predictive value) to evaluate diagnostic confidence, F1-score to balance precision and recall, and accuracy to reflect overall classification performance. Implementation Details: To ensure a fair comparison, we conducted five-fold cross-validation on each dataset, partitioning the labeled images into 80% training data and 20% test data. The training data was further divided into a training set and a validation set in a ratio 3 : 1, maintaining the original class distribution for hyperparameter tuning. To mitigate class imbalance in the training set, we employed the Random OverSampling Examples (ROSE) (Hayaty, Muthmainah, and Ghufran 2020) technique to balance the number of images in each class. Additionally, we applied dataaugmentation techniques, including image flipping, cropping, and scaling, to the training set to improve the model's generalization ability. For consistency, all retinal images were resized × 256 pixels.
All compared models are implemented using the Py-Torch framework. The segmentation components were pretrained on 933 samples from six public retinal vessel segmentation datasets: FIVES (Jin et al. 2022), DRIVE (Staal et al. 2004), STARE (Hoover, Kouznetsova, and Goldbaum 2000), CHASEDB1 (Budai et al. 2013a), HRF (Budai et al. 2013b), and Retinal Blood Vessel Segmentation (Wang et al. 2021b). The classification components were pre-trained on the ImageNet dataset, except for RETFound, which was trained on its custom dataset.
All experiments were accelerated using NVIDIA RTX A5000 GPUs. Model optimization was performed using the Adam optimizer. To enhance performance, we conducted a grid search to fine-tune key hyperparameters, including the learning rate, batch size, and disease class weight in the weighted cross-entropy loss function. The learning rate was explored within the range 5 × 10 -5 to 1 × 10 -3 , batch sizes were selected from {16, 32, 64, 128}, and class weight were varied from 0.5 to 0.9 with a step size of 0.1. We set the maximum number of training epochs to 200, with early stopping applied if validation performance did not improve within 20 epochs. The best-performing model checkpoint on the validation set was selected for testing.
Comparisons with Baselines: Table 1 presents the disease classification performance of all compared models across two datasets. Specifically, we have the following observations: SegImgNet consistently outperforms all baselines across key metrics on both datasets, demonstrating its superior capability to distinguish between disease and normal cases. While SegImgNet achieves slightly lower specificity (0.843 ± 0.042) and accuracy (0.857 ± 0.013) compared to VisionDeep-AI (0.885 ± 0.010 and 0.865 ± 0.010, respectively) on the e-ROP dataset, it remains highly competitive on these two metrics. More importantly, while VisionDeep-AI exhibits higher specificity and accuracy, it falls short in other critical metrics, particularly sensitivity (0.731 ± 0.028 for VisionDeep-AI vs. 0.831 ± 0.027 for SegImgNet). This lower sensitivity increases the risk of missed diagnoses, which can lead to delayed treatment. In medical applications, sensitivity is crucial, as missing a disease diagnosis can have far more severe consequences than misclassifying a healthy individual. Notably, SegImgNet achieves the highest sensitivity among all baselines on both data sets, confirming its effectiveness in clinical decision-making. Figure 3 shows the visualization of intermediate feature maps from the segmented image encoder of the top three models (SegImgNet, VisionDeep-AI and Multi-GlaucNet) across two datasets. We selected the feature maps produced by each model's second downsampling layer and visualized four representative channels, chosen based on their mean and variance. The visualization results demonstrate that our approach achieves higher structural clarity and consistency compared to the other two approaches. Specifically, SegImgNet more distinctly delineates prominent edges and anatomical structures, thereby enhancing its capability to preserve and highlight morphological features for accurate retinal analysis. Ablation Study: Here we investigated the contribution of each key component in SegImgNet, including the segmented image encoder, raw image encoder, and SGA block. Table 2 presents the performance of different model variants: "w/o segmented image encoder" excludes the segmented image encoder, "w/o raw image encoder" removes the raw image encoder, and "w/o SGA" omits the SGA block. The results demonstrate that each component is essential for optimal performance. Removing the segmented image encoder significantly reduces the model's ability to capture retinal structural features, while eliminating the raw image encoder weakens its capacity to extract global contextual information. Furthermore, the absence of the SGA block degrades classification performance, highlighting the importance of multi-scale retinal structural feature maps in enhancing representation learning. The complete SegImgNet model, incorporating all components, achieves the highest performance, emphasizing the importance of integrating local and global feature extraction with attention-based enhancement. These findings confirm that each module plays a critical role in maximizing disease classification accuracy.
In this study, we introduce SegImgNet, a deep learning model that integrates local retinal structural features from segmented images with global contextual information from raw images for disease classification. Extensive experiments on public and private datasets show that SegImgNet outperforms existing methods, demonstrating the effectiveness of segmentation-guided attention for feature enhancement. Our findings highlight the potential of incorporating retinal structural priors into deep learning frameworks to improve the robustness of AI-driven medical imaging. Future work will focus on optimizing feature fusion, expanding the model to broader ophthalmic applications, and improving generalization across diverse clinical datasets.