Background & Summary

Retinal image registration plays a pivotal role in disease diagnosis1,2, image-guided surgery3, monitoring of disease progression4, and image fusion5. The primary objective of image registration is to spatially align a query (source) image with a reference (target) image by determining a geometric transformation that accurately matches the corresponding features or structures between the two images. In the realm of ophthalmology, retinal image registration has become an indispensable tool, facilitating precise tracking of temporal changes in retinal anatomy and the alignment of different imaging modalities, which is crucial for evaluating disease evolution and treatment outcomes6,7.

In the field of ophthalmology, various imaging modalities are used, including color fundus (CF), optical coherence tomography (OCT), fluorescein angiography (FA). The integration of these diverse imaging modalities via registration enhances the comprehensive analysis of retinal pathologies. Numerous studies have been dedicates to proposing retinal image registration algorithms, thereby propelling the advancement of this domain. As delineated in several review papers6,7, these algorithms can be categorized into multi-modal and single modal image registration based on the input image modality. The corresponding datasets are summarized in Table 1, all of which come with the ground truth data for registration purposes. Datasets such as RODREP8 (http://www.rodrep.com/data-sets.html), TeleOphta9 (https://www.adcis.net/en/third-party/e-ophtha/), and VARIA10 (http://www.varpa.es/research/biometrics.html), which were originally created for application other than registration, are excluded from the table.

Table 1 The dataset and dataset review.
Full size table

Recent advancements in retinal imaging have led researchers to propose various multi-modal image registration datasets. For instance, Lee et al.11 introduced a private dataset in 2015, which includes CF and OCT images. In 2020, Almsasi et al.12 presented a FA-scanning laser ophthalmoscopy (FA-SLO) dataset, from patients with diabetic retinopathy. The PRIME-FP20 dataset13 (https://ieee-dataport.org/open-access/prime-fp20-ultra-widefield-fundus-photography-vessel-segmentation-dataset), introduced in 2021, comprises ultra-wide-field fluorescence angiography (FA) and fundus photography (FP) images. FOCTAIR14 (http://www.varpa.es/research/ophtalmology.html) contains FA and optical coherence tomography angiography (OCTA) images from retinal vein occlusion (RVO) patients. In 2024, MEMO15 (https://chiaoyiwang0424.github.io/MEMO/) was introduced, which consists of Erythrocyte-mediated Angiography(EMA) and OCTA images from normal patients.

While additional imaging modalities such as OCT and FA are employed as necessary for diagnostic or therapeutic purposes, color fundus photography is still the most frequently utilized data type in clinical settings. For example, FLoRI2116 (https://ieee-dataport.org/open-access/flori21-fluorescein-angiography-longitudinal-retinal-image-registration-dataset)includes ultra-wide-field fundus photography from 15 patients, presenting a significant challenge for registration due to its extensive field of view. The LSFG dataset17, from 2024, encompasses images from 15 patients with uveal melanoma. Given that color fundus photography remains the most commonly used imaging modality in hospitals, the FIRE dataset18 (https://projects.ics.forth.gr/cvrl/fire/) has been predominantly evaluated. It is categorized into three classes based on the extent of image overlap and the presence or absence of anatomical features. The majority of image pairs within the FIRE dataset are utilized for super-resolution and mosaicking purposes. Only 14 image pairs were acquired from different examinations of retinopathy, yet they exhibit significant overlap. All images in the FIRE dataset are of high resolution. With the integration of deep learning algorithms into fundus image registration, the accuracy of the dataset has already achieved a high level, with a reported 0.85 mean Area Under the Curve (mAUC) and RMSE less than 2.919. Consequently, in response to current demands, we construct a challenging monomodal fundus image registration dataset COph100, which contains 100 eyes and 491 pairs of images from Retinopathy of Prematurity (ROP) infants. Given the current publicly available ROP datasets1 (https://doi.org/10.6084/m9.figshare.c.6626162.v1), this paper primarily discusses how to utilize these existing public datasets to further pursue research of one’s interest.

To the best of our knowledge, the COph100 dataset is the first retinal registration dataset specifically focused on disease progression in infants. Its potential impact, which addresses several existing limitations, can be outlined as follows: First, Minimal apprarance variablity: A common limitation of most retinal image registration datasets is the lack of significant appearance variation, which does not adequately reflect the diversity encountered in clinical practice. The COph100 dataset, however, includes image pairs with substantial appearance differences resulting from variations in acquisition time, patient condition, and imaging environments. This diversity enhances the generalizability of registration algorithms, enabling them to perform more robustly in real-world clinical settings where image characteristics vary widely. Second, Pediatric pathologies focus: Existing datasets primarily focus on adult retinal pathologies, thereby neglecting the distinct features and challenges of pediatric retinal diseases. The COph100 dataset addresses this gap by concentrating on pediatric retinal pathologies, expanding the clinical relevance and applicability of registration algorithms beyond the adult population. Third, Scalability and population diversity: Current retinal image registration datasets are limited in scale, often involving fewer than 52 eyes, which can introduce bias in algorithm performance. In contrast, the COph100 dataset offers greater scalability and includes a more diverse patient population. This increased diversity provides a more robust basis for developing algorithms that are generalizable across different demographic groups, leading to improved clinical outcomes. Therefore, our proposed COph100 dataset represents a significant advancement in retinal image registration by providing a more diverse, pediatric-focused, and scalable dataset that improves the robustness and clinical applicability of registration algorithms.

Methods

Data Preparation

Image datasets

In this study, we aim to utilize publicly available retinal datasets to construct a comprehensive retinal image registration dataset. Our first step involves identifying datasets that include image pairs, varying poses or follow-up examinations, such as RODREP8, TeleOphta9, and a dataset proposed by Timkovic et al.1 (https://doi.org/10.6084/m9.figshare.c.6626162.v1). Certain datasets, like the 2021 Retinopathy of Prematurity (ROP) dataset20, which only contains images from different poses, are not considered for this study. We compare these dataset with FIRE, as shown in Table 2. The RODREP dataset, proposed in 2015, consists of fundus images from 70 diabetic patients with two examinations. TeleOptha, introduced in 2013, includes images from one or both eyes of healthy and diabetic retinopathy patients, with one or two examinations. The dataset proposed by Timkovic et al.1 in 2024 contains 2-9 examinations of premature infants with ROP. It includes images captured at three different resolutions by various devices, providing a diverse set of eyes. The image resolution for RODREP, TeleOphtha and two sections (1240 × 1240 and 1440 × 1080) of the Timkovic et al. dataset1 is relatively high.

Table 2 Preparation of datasets.
Full size table

Several factors were considered in selecting a suitable dataset for our challenging COph100 dataset. First, the dataset should include a large number of patient eyes. Second, it should contain images from patients with at least two examinations. Finally, the image resolution should not be excessively high, as high-quality retinal images are not always obtainable in clinical practice. Based on these criteria, we have decided to construct the challenging COph100 dataset using the lower-resolution images (640 × 480) from the dataset proposed by Timkovic et al.1.

Inclusion-exclusion criteria

Our study primarily focuses on image registration across multiple examinations, aiming to analyze disease progression over time. The specific selection criteria are outlined in Fig. 1. The original ROP grading dataset includes 396 eyes from 188 patients. To construct a more challenging registration dataset than FIRE, we deliberately selected images with greater appearance differences, such as variations in image resolution, illumination, compared to those in the FIRE dataset. Additionally, to assess disease progression effectively, we excluded patients with only a single examination, ultimately narrowing the dataset down to 118 eyes.

Fig. 1
figure 1

Summary diagram of inclusion-exclusion criteria.

Full size image

Capturing retinal images from infants, unlike adults, poses significant challenges. Infants cannot follow instructions such as maintaining fixation on a point or keeping their eyes still, which complicates the process of obtaining high-quality fundus images. The resulting images often suffer from large occlusions caused by diseases or eyelids, poor focus, or significant rotation and translation between examinations, leading to limited image overlap. Furthermore, since physicians need to monitor changes in lesion areas over time during diagnosis, we made efforts to select image pairs that consistently include the lesion area at different time points. This selection process is intended to facilitate future analysis of disease progression. Ultimately, we obtained 100 eyes with 491 image pairs for our COph100 dataset.

Data processing

Registration Groundtruth

In image registration research, the evaluation of methods is typically conducted using both quantitative and qualitative metrics. Quantitative assessments often rely on either manually designated or automatically detected control points, which enable a concise representation of registration error as a single numerical value. Other quantitative methods, such as those that compute a transformation matrix, express error in terms of multiple parameters. On the other hand, qualitative evaluations focus primarily on the visual inspection of vessel alignment within the images, which necessitates expert input and lacks the ability to provide objective, quantitative comparisons across different registration techniques.

In the proposed COph100 dataset, we offer the actual corresponding points needed to compute registration errors, which are termed as control points, as illustrated in Fig. 2. For each eye, the location of a control point j in the image of first examination is denoted as E1j, and the corresponding point in the image of other examination i as Eij. The registration process uses the point Eij as input and transforms it to the new set of coordinates referred to as Mij. Thus, the Mij coordinates represent the post-registration positions of the Eij points. Ideal registration results in the points Mij and Eij being indistinguishable, with a pixel distance of 0 between them. In Fig. 2, each point corresponds to each other in all 9 examinations. The ground truth points are primarily marked around the vessel intersections.

Fig. 2
figure 2

Image from patient with 9 examinations. The challenges include blur, obstruction, large lesions, illumination and color changes.

Full size image

Retinal Vessel Segmentation

The vessel structure plays a crucial role in analyzing the condition of the retina. To extract the vessel structure, we developed a vessel segmentation model based on SS-MAF21 (https://github.com/Qsingle/imed_vision). Following prior research22, we selected the green channel from the RGB image format as the input for the model, and we trained the model using the publicly available FIVES dataset23 (https://doi.org/10.6084/m9.figshare.19688169.v1). The training settings were consistent with21, where the 600 images in the training dataset were split with a 4:1 ratio for training and validation purposes. Upon training, the model achieved performance metrics of 91.56% in Dice score, 89.81% in sensitivity (SE), and 89.32% in bookmaker informedness (BM) at the test set of the FIVES dataset. The segmentation results are visualized in Fig. 3, demonstrating that the model successfully captures nearly all of the vessel structures. The collaborating opthalmologists also evaluated the segmentation results and concluded that they are sufficient for disease progression analysis. Since the original grading dataset proposed by Timkovic et al.1 includes a section on lesion segmentation, we did not extract the lesion areas separately in our study.

Fig. 3
figure 3

Visualization samples of Retinal vessel segmentation results.

Full size image

Formulations for the evaluation metrics are as follows:

$$Dice=\frac{2\times TP}{2\times TP+FN+FP},SE=\frac{TP}{TP+FN},BM=\frac{TP}{TP+FN}+\frac{TN}{TN+FP}-1$$
(1)

where TP, TN, FP, and FN are the true positive, true negative, false positive, and false negative respectively.

Data Records

The files available on Figshare24 (https://doi.org/10.6084/m9.figshare.27061084) and Github (https://gaiakoen.github.io/yanhu/research/Retinal_Image_Registration) include image folders, a data summary in xlsx format and a Python script for extracting corresponding images from the original dataset. Our COph100 dataset stands apart from the original classification dataset1 in three significant ways: Firstly, it is tailored for image registration to monitor disease progression, rather than for classification. Secondly, we have carefully selected a subset of images (325 out of 6,004) from the original dataset, following strict inclusion-exclusion criteria. Finally, we provide 10 point pairs for each image pair, complete with segmented vessel data, making this the largest publicly available retinal image registration dataset for infants to date.

The published dataset comprises 100 eyes, with a total of 491 pairs of images. The details of the examinations are listed in Table 3. There are 33 eyes containing 2 examinations, 43 eyes containing 3 examinations, 14 eyes containing 4 examinations, and so on. For the folder name, the folders labeled ‘003’ and ‘003-1’ indicate that we selected two eyes from the same patient. The folder labeled ‘04’ suggests that one of the patient’s eyes was not suitable for registration according to our inclusion and exclusion criteria. The structure of the files within our COph100 dataset is illustrated in Fig. 4. Taking the ‘005’ folder as an example, each patient undergoes three examinations, resulting in three distinct images. For each image, the corresponding data includes point pairs (stored in .json files), segmented vessel data (stored in .nii.gz files), vessel masks (stored in .png files), and vessel overlay images (also stored in .png files).

Table 3 The distribution of examination sessions within our COph100 dataset.
Full size table
Fig. 4
figure 4

The details of our COph100 dataset files.

Full size image

Technical Validation

Image quality evaluation

In order to compound the complexity of image registration tasks, certain artifacts are present in fundus images, which can significantly impact the training phase of registration algorithms. Research on color fundus image quality assessment exists25,26,27. However, these algorithms cannot be applied to our dataset due to unavailable code, privacy constraints, or differences between their datasets and our infant images. Consequently, our research team, in collaboration with professional ophthalmologists, has undertaken the evaluation of image quality. This study focuses on three primary challenges: (1) Blur, which is induced by eye movement or out-of-focus conditions; (2) Obstruction, caused by occlusion from disease-related lesions, hemorrhages, patient non-cooperation, probe rotation issues during capture, or turbid refractive media; and (3) Changes, encompassing disease progression, anatomical variations in the visual field, shifts in illumination and color, as well as spatial overlap.

In our dataset of 100 eyes with 325 images, we assessed the quality of each image with respect to blur and obstruction, as well as variations within each eye. Many images presented with multiple challenges. Figure 5 provides statistics and examples illustrating the issues of blur and obstruction present in our dataset. Out of the 325 images, 213 images were affected by obstruction, and 100 images suffered from blur. A total of 82 images experienced both challenges simultaneously, with only 21 images being relatively clear. Consequently, we believe that overcoming the low accuracy of registration caused by obstruction and blur will be an important research challenge for the future. Figure 5(a),(b) depict images obstructed by turbid refractive media and hemorrhages, respectively. Figure 5(c) represents images affected by both blur and obstruction, while Fig. 5(d) illustrates the challenge of blur alone.

Fig. 5
figure 5

Statistics and examples of the challenge involving blur and obstruction.

Full size image

Figure 6 presents the statistics and examples of the challenges related to changes observed in our COph100 dataset. Over half of the eyes exhibit various changes. In Fig. 6(a), the areas of obstruction differ between the two images, and there are notable rotational changes between them. In Fig. 6(b), the regions marked by arrows show differences. Prominent hemorrhages are present in the left image during the first examination, but they have almost resolved in the right image by the third examination.

Fig. 6
figure 6

Statistics and examples of the challenge involving changes.

Full size image

Registration evaluation

To assess the registration performance of existing methods on our dataset, we follow the approach outlined in paper28 and report on three categories of results: the ratios of failed, inaccurate and acceptable registrations. For each pair of images in our dataset, the early examination image Iq is used as the query image, while the late examination image acts as the reference Ir. A registration attempt is classified as failed if the number of feature matches is less than 4. For successful registrations that obtain a homography, the quality is evaluated using two metrics: the Median Error (MEE) and the Maximum Error (MAE) of the distances between the matched points Mq in the query image and their corresponding estimated positions Er in the reference image. Registrations are deemed acceptable if both MEE is less than 20 and MAE is less than 50; otherwise, they are labeled as inaccurate. In addition to these measures, we report the Root Square Error (RMSE) for corresponding points after transformation within the acceptable category. Furthermore, we include the Area Under the Curve (AUC) metric18 to provide a comprehensive reflection of each method’s overall performance.

To comprehensively assess the performance of existing algorithms on our dataset, we conducted experiments based on traditional machine learning and deep learning based algorithms. Traditional algorithms include SIFT29, GDB-ICP30 (https://vision.cs.rpi.edu/gdbicp/exec/), REMPE31 (https://projects.ics.forth.gr/cvrl/rempe/). Deep learning-based algorithms include a multitude of algorithms trained on natural scene images (Superpoint32, GLAMpoints28, R2D233, SuperGlue34, LoFTR35) and specifically trained on fundus images (SuperRetina36, Swin U-SuperRetina37, LK-SuperRetina37, and SuperJunction19). We adopted the same testing method as the FIRE dataset, which involves using all the pre-trained models from the official platforms to test on the COph100 dataset.

Moreover, in light of the recent focus on vascular information in image registration studies, we have adopted a similar approach in our experiments. We utilize the segmentation results of image pairs as the query and reference images to specifically assess the performance of existing registration methods on vascular segmentation images. This choice allows us to evaluate the methods’ effectiveness in contexts where vascular alignment is crucial. Note that all our evaluations are conducted at the original image resolution of 640 × 480 pixels.

Table 4 summarizes the test results, where the values inside and outside the parentheses represent the results with and without segmentation information, respectively. For traditional methods, GDB-ICP demonstrates strong performance with the lowest failure rate (0%) and an inaccuracy rate of 22.20% when no segmentation is applied. It further enhances its performance on vascular segmentation images, achieving a low failure rate of 1.22% and decreasing the inaccuracy rate to 2.24%. When segmentation is used, its performance even surpasses that of the deep learning-based methods. SIFT shows a moderate improvement with segmentation, reducing both the failure rate (from 0.61% to 0%) and the inaccuracy rate (from 86.35% to 59.47%). In contrast, the REMPE method, which depends on 3D modeling of the fundus for registration, fails completely when segmentation information is used, as the segmentation data does not support the necessary 3D modeling process.

Table 4 Experiment results without (with) segmentation mask based on existing traditional machine learning and deep learning based methods.
Full size table

Deep learning methods such as SuperPoint, SuperGlue, LoFTR, and SuperJunction have delivered impressive results across all metrics. In contrast, methods including SuperRetina, Swin U-SuperRetina, and LK-SuperRetina have shown higher failure rates and inaccuracies, leading to poor overall performance. When segmentation results are incorporated, the performance of these models further declined, suggesting that models trained exclusively on the fundus dataset may lack robustness when applied to different datasets. Generally, methods trained in natural image environments have demonstrated improved accuracy with the addition of segmentation, except for SuperGlue. This discrepancy might be attributed to the fact that segmentation results can reduce the distinctiveness of descriptors in SuperGlue, which hinders effective matching. We noted that segmentation information tends to improve the registration accuracy for most methods, which underscores the potential benefits of exploring new strategies that more effectively integrate vascular segmentation data to enhance the performance of fundus image registration tasks. In Fig. 7, we present the registration results using a checkerboard mosaic visualization. It is evident that SIFT, GLAMpoints, and SuperRetina clearly failed. Even the other methods, despite performing better, still exhibit noticeable misalignments in the vascular structures compared to the ground-truth registration results. This highlights the need for registration methods to further improve accuracy and achieve complete alignment of the blood vessels in the images for more reliable outcomes. Therefore, the results from the table indicate that our proposed COph100 dataset represents certain challenges for both traditional machine learning and deep learning methods. There is considerable potential for enhancement in the future.

Fig. 7
figure 7

Mosaic visualization of registration results. (a) SIFT (b) GDB-ICP (c) REMPE (d) SuperPoint (e) GLAMpoints (f) R2D2 (g) SuperGlue (h) LoFTR (i) SuperRetina (j) Swin U-SuperRetina (k) LK-SuperRetina (l) SuperJunction.

Full size image

Usage Notes

The COph100 dataset is intended to serve as a resource for researchers working in the field of medical image analysis, particularly those interested in image registration, disease progression analysis. The COph100 dataset presented in this paper can be downloaded through the link mentioned above.