Introduction
Retinopathy of prematurity (ROP) is a vasoproliferative disease of the retina that predominately affects infants with low birth weight and low gestational age. Through clinical trials, including the Cryotherapy for ROP and Early Treatment for ROP (ET-ROP), a consensus policy statement was established for the screening and management of ROP. Despite these evidence-based guidelines and policy statements, ROP continues to be a leading cause of childhood blindness across the world. Even as the global burden of ROP continues to rise, ROP care is limited by the workforce shortage of ophthalmologists who are adequately trained in ROP management.
In the United States, only 11% of ophthalmologists manage ROP, and this cohort is predicted to become smaller. A recent survey noted that nearly 20% of ophthalmologists who manage ROP would like to stop treating this ocular disease. This workforce shortage may also be exacerbated by the limited exposure to ROP during residency and fellowship training.
The complexity of ROP is greater when examined on a global level. Although there have been advancements in neonatal care in middle-income countries, shortage of competent staff, lack of sufficient medical equipment, and inadequate protocols for the establishment of structured screening programs have led to a rise in children at risk for blindness. As we are now facing the “third epidemic” of ROP, middle-income countries are faced with higher rates of preterm births, as well as higher rates of admission to neonatal intensive care units.
With the expansion of Internet capabilities and access, tele-education programs may be implemented to enhance trainee knowledge in diagnostic criteria, as well as increase the volume of trained providers focused on ROP care. Previous work has characterized the utility of a tele-education system for ophthalmology trainees in the United States. However, little data exist on evaluation of ophthalmology training programs in the international context. Participation in the Global Education Network for Retinopathy of Prematurity (GEN-ROP) tele-education system has demonstrated improvement in United States and international trainees' sensitivity and specificity in diagnosis of ROP based on pretest and posttest analyses. The purpose of this study was to describe errors in ROP diagnosis made by ophthalmologistsin-training from programs in middle-income countries including Brazil, the Philippines, and Mexico who participated in a tele-education program.
Methods
The Weill Cornell Medical College Human Studies Committee prospectively approved all aspects of the use and analysis of retinal images and educational material used in this study. Administration of the web-based system was also reviewed by The Weill Cornell Medical College Human Studies Committee. This was considered to be research in an established or commonly accepted educational setting involving normal educational practices, such as research on the effectiveness of instructional techniques, curricula, and instructional strategies. The research adhered to the tenets of the Declaration of Helsinki.
Case Acquisition
This study is a subanalysis of a tele-education system designed to improve trainees' competency in ROP diagnosis and has been previously described in detail. In summary, from a repository of 2,500 web-based cases, 16 unique cases consisting of wide-field retinal images were selected, of which 4 cases were repeated for a total of 20 cases.
Study Participants and Study Design
Ophthalmologists-in-training from training programs in Brazil, Mexico, and the Philippines were recruited by the coauthors (KEJ, RVPC, RGASA, CVV, MAM-C). Trainees from all postgraduate training years were allowed to participate.
Trainees were directed to a secure web-based server and were prompted with a test consisting of the 20 cases (Figure 1). Each case displayed 2 eyes, for a total of 40 eyes with varying levels of ROP to be evaluated. The trainees were asked to input the diagnosis, accounting for all features of ROP, in accordance with the internationally standardized ROP classification system: plus disease (none, pre-plus, plus), zone (I, II, III), stage (1, 2, 3, 4, 5), and category (none, mild, type 2 ROP or pre-plus, treatment-requiring), and aggressive posterior ROP (yes, no) for each eye. Throughout the assessment, trainees could review but not change responses to completed cases. Feedback was not given. Modules were available for use in Medical English and Portuguese.
Figure 1
Summary of study methods for retinopathy of prematurity (ROP) competency evaluation. Two hundred ophthalmologists-in-training from middle-income countries were directed to a web-based system. They were prompted with 20 ROP cases and were asked to indicate the presence of plus disease, zone, stage, and category of ROP. The trainees' performance was assessed in accordance with a consensus reference standard diagnosis that was determined for each case. The rates at which common errors occurred were statistically analyzed.
Consensus Reference Standard Diagnosis
The methods for setting the reference standard diagnosis have been previously described. In short, three ROP experts evaluated the color fundus photographs for each case and determined the ROP diagnosis. In addition, an ROP expert performed indirect ophthalmoscopy on the infant and determined the diagnosis. Based on the combined expert findings via the two aforementioned methods, a consensus reference standard diagnosis was developed for each case.
Data Analysis
The participants' responses recorded for each case were analyzed using statistical software (IBM SPSS 25 Statistics; SPSS, Inc). A two-sided P value of less than .05 constituted statistical significance. Univariate relationships between the error rates among the cohorts and the different countries were evaluated without adjustment for other covariates using the two-sample t test, Wilcoxon ranked-sum test, or Spearman's correlation for continuous variables and Fisher's exact test for categorical variables. Diagnostic performance of the participants with respect to the consensus reference standard diagnosis was measured.
Results
Table 1 summarizes key characteristics of the 200 ophthalmologists-in-training who participated in the study.
Diagnostic Error Rate of ROP Category Diagnosis
For all categories of ROP, the diagnostic error rate, defined as the percentage of incorrect diagnosis, was 50% (3,567 of 7,066 responses) across all countries: 48% (430 of 896 responses) for country A, 59% (922 of 1,566 response) for country B, and 48% (2,215 of 4,604 responses) for country C. The diagnostic error rate for the diagnosis of treatment-requiring ROP was 45% (76 of 168 responses) for country A, 43% (126 of 294 responses) for country B, and 32% (290 of 918 responses) for country C. The diagnostic error rate for the diagnosis of type 2 ROP was 74% (206 of 280 responses) for country A, 77% (379 of 490 responses) for country B, and 79% (972 of 1,232 responses) for country C.
Figure 2 shows the diagnostic error rate of category that is stratified by country and postgraduate year (PGY) for the ophthalmologists-in-training. For category diagnosis, PGY-1 trainees had a mean error rate of 64%, which was significantly higher than PGY-2 trainees with a mean error rate of 52% (mean difference: 12.1; 95% CI: 2.9 to 21.1; P = .021) and PGY-3 trainees with a mean error rate of 46% (mean difference: 18.0; 95% CI: 10.0 to 26.0; P = .003). There was no significant difference in error rates for zone between the 3 years.
Figure 2
Diagnostic error rate of retinopathy of prematurity (ROP) category among ophthalmologists-in-training stratified by postgraduate year. For each web-based case, trainees were prompted to provide the category of ROP (none, mild, type 2 or pre-plus, treatment-requiring). The rates at which the category of ROP was misdiagnosed by ophthalmology trainees were stratified by postgraduate year (PGY-1, PGY-2, PGY-3) and by country (A, B, C).
Figure 3 shows the distribution of responses among all ophthalmologists-in-training for each diagnostic category of ROP. The error rate was highest for clinical cases with a consensus reference standard diagnosis of type 2 or pre-plus ROP (1,557 of 2,002 responses, 77%), with a trend toward overdiagnosis (27% underdiagnosis vs 50% overdiagnosis; mean difference: 23.4; 95% CI: 12.1 to 34.7; P = .005). Furthermore, in cases with a consensus reference standard diagnosis of treatment-requiring ROP, trainees reported a diagnosis of no ROP in 6.1% of responses, mild ROP in 11.7% of responses, and type 2 or pre-plus in 17.8% of responses.
Figure 3
Distribution of responses among all ophthalmologists-in-training for each diagnostic category of retinopathy of prematurity (ROP). Trainees were prompted to provide the category of ROP (none, mild, type 2 or pre-plus, treatment-requiring) for each case presented. The correct diagnostic category was determined in accordance with a consensus reference standard diagnosis. The trainees' responses are represented proportionately for each category of ROP.
Reasons for Incorrect ROP Diagnosis
Table 2 displays the incorrect category diagnosis of the ophthalmologists-in-training in comparison to the error rates of plus disease, zone, and stage diagnoses. Misdiagnosis of treatment-requiring ROP as type 2 ROP was most commonly associated with incorrectly identifying plus disease (plus disease error rate = 18% with correct category diagnosis vs 69% when misdiagnosed; mean difference: 51.0; 95% CI: 49.3 to 52.7; P = .003). When treatment-requiring ROP was misdiagnosed as type 2 ROP, there was no significant association with the incorrect identification of zone (zone disease error rate = 62% with correct category diagnosis vs 65% when misdiagnosed, mean difference: 2.9; 95% CI: −6.9 to 1.98; P = .472) or stage (stage disease error rate = 45% with correct category diagnosis vs 64% when misdiagnosed; mean difference, 18.6; 95% CI: −4.83 to 42.83; P = .061).
Similarly, overdiagnosis of type 2 ROP as treatment-requiring ROP was most commonly associated with incorrectly identifying plus disease (plus disease error rate = 36% with correct category diagnosis vs 74% when misdiagnosed; mean difference: 38.1; 95% CI: 25.48 to 50.52; P = .0011)
Figure 4 shows a sample case of a common mis-diagnosis in which the consensus reference standard diagnosis was zone I, stage 3 ROP with plus disease but the ophthalmologists-in-training most commonly provided a diagnosis of zone II, stage 3 ROP and pre-plus disease.
Figure 4
Eye examination with diagnostic discrepancies between the consensus reference standard diagnosis and trainee diagnosis. The patient is a 23-week-old infant with a birth weight of 608 grams and a postmenstrual age of 35 weeks at the time of imaging. The consensus reference standard diagnosis was zone I, stage 3 retinopathy of prematurity (ROP), and plus disease. The trainees most commonly provided a diagnosis of zone II, stage 3 ROP and pre-plus disease.
Discussion
This study characterizes errors in ROP diagnosis made by ophthalmologists-in-training from programs in three middle-income countries. The key findings from this study are: (1) ophthalmologists-in-training in middle-income countries misdiagnosed any form of ROP approximately half of the time and (2) incorrect diagnosis of plus disease was the most common factor leading to misdiagnosis of clinically significant ROP (type 2 ROP or treatment-requiring ROP).
The first key finding is that ophthalmologistsin-training, from all three countries, misdiagnosed all categories of ROP between 48% and 58% of the time. There was no significant difference in the error rates of ROP diagnosis among the three countries. PGY-1 trainees had a mean error rate of 64%, which was significantly higher than PGY-2 (52%) or PGY-3 trainees (46%) (Figure 2). The high rates of misdiagnosis may be due to insufficient training on classifying the different type ROP categories (none, mild, type 2 or pre-plus, treatment-requiring). In our study, ophthalmologists-in-training misdiagnosed treatment-requiring ROP as type 2 or pre-plus disease, yet 31% of those same trainees correctly identified plus disease, which was present in each of the cases (Table 2). These findings may highlight the insufficiencies in understanding the definitions of treatment-requiring ROP and type 2 ROP.
The second key finding is that incorrect identification of plus disease was among the most common errors, especially in cases with clinically significant ROP. Misdiagnosis of treatment-requiring ROP as type 2 ROP was most commonly associated with incorrectly identifying plus disease (plus disease error rate = 18% with correct category diagnosis vs 69% when misdiagnosed, P = .003). Similarly, cases with a consensus reference standard diagnosis of type 2 or pre-plus ROP were significantly overdiagnosed in 1,009 of 2,002 total responses (50%). The overdiagnosis of type 2 ROP as treatment-requiring ROP was most commonly associated with incorrectly identifying plus disease (plus disease error rate = 36% with correct category diagnosis vs 74% when misdiagnosed, P = .0011).
The variability in diagnosis of these categories of ROP is important from a clinical perspective. In cases of treatment-requiring ROP, intervention (laser photocoagulation, anti-vascular endothelial growth factor therapy, and/or surgery) should be performed within 48 to 72 hours of detection. In cases of type 2 or pre-plus disease, infants should be monitored closely for progression to treatment-requiring disease. In the ET-ROP study, of 294 eyes with a diagnosis of type 2 ROP at the first visit, 65 (22.1%) progressed to type 1 or treatment-requiring ROP. Overdiagnosing type 2 or pre-plus disease as treatment-requiring ROP may lead to unnecessary treatments and exposure because this disease category may regress without intervention.
The discrepancies in the diagnosis of clinically significant ROP by ophthalmologists-in-training has previously been documented: among 31 ophthalmologists-in-training from the United States and Canada, there was 49% sensitivity for plus disease diagnosis and 48% sensitivity for the diagnosis of treatment-requiring ROP. Furthermore, in a study of five pediatric ophthalmology fellows from the United States, type 2 ROP was accurately diagnosed by all trainees less than 36% of the time, and the identification of plus disease was noted as a reason for discrepancy in 40% of cases. Similarly, in a study of seven retina fellows from the United States, type 2 ROP was misdiagnosed as treatment-requiring ROP in 47% of responses, and the identification of plus disease was noted as a reason for discrepancy in 40% of cases. These findings suggest that ophthalmologists-in-training at both the resident and fellow level in middle-income and high-income countries have similar discrepancies in the diagnosis of clinically significant ROP. Moreover, variability in diagnosis among ROP experts has been demonstrated in the literature. Ryan et al explored inter-reader variability among ROP experts when developing a reference standard diagnosis. They found that diagnoses varied between readers when using fundus images, and in addition, variability was noted between image-based diagnosis and that made by indirect ophthalmoscopy. However, it was demonstrated that the use of multiple readers reduced the variability and allowed for a consensus to be reached. In this aforementioned study by Ryan et al, there was overall consensus among the three experts 73% of the time with respect to the clinical examination. When developing the reference standard diagnosis (using the fundus images and indirect ophthalmoscopy), overall agreement among the experts occurred at a rate of 81% with a weighted kappa value of 0.751.
Plus disease is a hallmark for the diagnosis of ROP requiring treatment and failure to recognize plus disease appropriately can lead to irreversible blindness secondary to progression of ROP. The variability in plus disease diagnosis has been well documented among both ophthalmologistsin-training and ROP experts. Novel machine learning paradigms have been developed to improve diagnostic accuracy in ROP. Brown et al reported findings on the i-ROP Deep Learning system that showed high accuracy for detecting plus disease from wide-angle posterior pole retinal images through a fully automated algorithm. More recently, a quantitative vascular severity score was developed that could be applied to identify diagnostic categories, overall disease severity, and monitor disease progression and regression over time in an automated fashion.
It has also been demonstrated that the i-ROP Deep Learning system can predict the stage of ROP with high diagnostic accuracy. High performance by the current model is associated with the use of a training dataset that is characteristically similar to the population being tested. Diversifying the training dataset may subsequently broaden the applicability of the system. The i-ROP Deep Learning system has been trailed using data from Nepal, Mongolia, and India and has demonstrated its potential to promote accurate, automated diagnostic assessments. Notably, the i-ROP Deep Learning system received breakthrough status by the U.S. Food and Drug Administration for use as a device in a clinical setting.
Given that incorrect plus disease diagnosis was the most common factor leading to misdiagnosis of clinically significant ROP, machine learning algorithms may serve as an important adjuvant for improving ROP education for ophthalmologistsin-training, particularly in regions where training is insufficient or unavailable. Artificial intelligence systems may not only assist in automatically identifying the correct diagnosis, but may also aid in generating targeted educational interventions for each user to improve diagnostic outcomes.
In parallel to these advancements, tele-education initiatives may play a key role in improving health care education for ROP. The lack of human resources is perhaps the most critical factor to limit improvements in health care education and delivery in middle-income countries. Compounding the human resource problem, among medical trainees in middle-income countries, access to quality medical education is limited, which hinders adequate training in the management of a complex disease such as ROP. Even in the United States, trainees generally do not feel well qualified to perform ROP care independently. Because ROP training is typically a component of vitreoretinal and pediatric ophthalmology fellowships, which are not under the direction of the Accreditation Council for Graduate Medical Education, there are no recommendations for screening and surgical volumes or for the fulfillment of core competencies.
In many middle-income countries where the number of cases of ROP is rising, the current medical education system still faces challenges in training enough qualified providers to manage the emerging epidemic. Furthermore, screening protocols may often be inconsistently implemented and some protocols may not be designed to ensure that every child at risk for severe ROP is evaluated. With the expansion of Internet access in middle-income countries, tele-education systems that can disseminate contextually appropriate screening protocols and demonstrate significant improvement in diagnostic accuracy have the potential to greatly impact the quality of ROP care and increase the volume of trained providers focused on ROP care.
This study is a subanalysis of a previous study that demonstrated improvement in ophthalmology trainees' diagnostic performance in ROP after participation in the tele-education system. Similar tele-education methods have been piloted among medical students to improve detection of diabetic retinopathy, and use of the system as an educational resource was well received among participants. Tele-education has the potential to equip providers with the skills needed to screen for ROP in regions lacking specialists, as well as to train ophthalmologists to provide remote screening via image-based telemedicine. Future studies are warranted to delineate the necessary qualifications for a provider, including non-ophthalmologists, to conduct remote reading. Moreover, a standardized method for obtaining these qualifications to become a remote reader needs to be established, and a tele-education system is one means by which this can be achieved. Particularly in the era of coronavirus virus disease 2019 (COVID-19), there is an increased role for telemedicine, and it is paramount that providers gain expertise in navigating health care delivery through telemedical methods. It is especially important to have well-trained providers performing tele-triaging to ensure acute disease presentations are not missed. These competencies for practicing telemedicine can be attained through formal web-based training.
There are several limitations to this study. Although our sample size of 200 participants representing trainees from three different countries may allow for generalizability of the data, the majority of those participants (134 of 200 trainees; 64%) were recruited solely from country C. Individuals from country C may have received training that was fundamentally different from the other cohorts; however, country-specific analyses of the data suggest consistent results across countries. Trainees' diagnostic skills were assessed solely on a web-based platform using color fundus photographs and demographic data. Deficiencies in clinical diagnosis and screening using indirect ophthalmoscopy were not assessed. However, because clinical practice has evolved to include color fundus photographs as part of the clinical evaluation, it is important to characterize trainees' diagnostic competency when using imaging. In addition, the trend toward using telemedicine in ophthalmology relies on the ability to accurately reach a diagnosis solely based on historical information and digital imaging. The language barrier may have hindered the trainees' ability to comprehend the online modules. Misunderstanding of terms when completing case forms could potentially account for trainees committing inadvertent diagnostic errors. However, a Portuguese module was developed for this study for one of the countries, and country-specific analyses of the data suggest consistent results across countries.
The findings of this study contribute to our knowledge on the competency of ROP diagnosis by ophthalmologists-in-training from middle-income countries. The common errors characterized in this study may promote curricular changes and aid in efforts to standardize ROP training. In addition, with the introduction of a revised international classification of ROP, web-based educational programs will be beneficial for ROP examiners worldwide.
References
- 1. Cryotherapy for Retinopathy of Prematurity Cooperative Group. Multicenter trial of cryotherapy for retinopathy of prematurity: preliminary results. Pediatrics. 1988;81(5):697–706–. PMID:
- 2. Early Treatment For Retinopathy Of Prematurity Cooperative Group. Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial. Arch Ophthalmol. 2003;121(12):1684–1694–. PMID:
- 3. Gilbert C. Retinopathy of prematurity: a global perspective of the epidemics, population of babies at risk and implications for control. Early Hum Dev. 2008;84(2):77–82–. PMID:
- 4. Wong RK, Ventura CV, Espiritu MJ, et al. Training fellows for retinopathy of prematurity care: a Web-based survey. J AAPOS. 2012;16(2):177–181–. PMID:
- 5. Nagiel A, Espiritu MJ, Wong RK, et al. Retinopathy of prematurity residency training. Ophthalmology. 2012;119(12):2644–5.e1–, 2. PMID:
- 6. Chan RV, Patel SN, Ryan MC, et al. The Global Education Network for Retinopathy of Prematurity (Gen-Rop): development, implementation, and evaluation of a novel tele-education system (an American Ophthalmological Society Thesis). Trans Am Ophthalmol Soc. 2015;113:T21–T226–. http://www.ncbi.nlm.nih.gov/pubmed/26538772.
- 7. Millán T, de Carvalho KM. Satisfaction with ophthalmology residency training from the perspective of recent graduates: a cross-sectional study. BMC Med Educ. 2013;13(1):75. PMID:
- 8. Ajay K, Krishnaprasad R. Feedback of final year ophthalmology postgraduates about their residency ophthalmology training in South India. Indian J Ophthalmol. 2014;62(7):814–817–. PMID:
- 9. Campbell JP, Swan R, Jonas K, et al. Implementation and evaluation of a tele-education system for the diagnosis of ophthalmic disease by international trainees. AMIA Annual Symposium Proceedings AMIA Symposium. 2015;2015:366–375–.
- 10. Fierson WM, Saunders RA, Good W, et alAmerican Academy of Pediatrics Section on OphthalmologyAmerican Academy of OphthalmologyAmerican Association for Pediatric Ophthalmology and StrabismusAmerican Association of Certified Orthoptists. Screening examination of premature infants for retinopathy of prematurity. Pediatrics. 2013;131(1):189–195–. PMID:
- 11. Fielder AR, Wilkinson AR. The further refinement of ROP treatment. commentary on the early treatment of retinopathy of prematurity study: structural findings at 2 years of age. Br J Ophthalmol. 2007;91(2):136–137–. PMID:
- 12. Myung JS, Paul Chan RV, Espiritu MJ, et al. Accuracy of retinopathy of prematurity image-based diagnosis by pediatric ophthalmology fellows: implications for training. J AAPOS. 2011;15(6):573–578–. PMID:
- 13. Paul Chan RV, Williams SL, Yonekawa Y, Weissgold DJ, Lee TC, Chiang MF. Accuracy of retinopathy of prematurity diagnosis by retinal fellows. Retina. 2010;30(6):958–965–. PMID:
- 14. Ryan MC, Ostmo S, Jonas K, et al. Development and evaluation of reference standards for image-based telemedicine diagnosis and clinical research studies in ophthalmology. AMIA Annual Symposium Proceedings AMIA Symposium. 2014;2014:1902–1910–. https://pubmed.ncbi.nlm.nih.gov/25954463https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4419970/.
- 15. Ataer-Cansizoglu E, Bolon-Canedo V, Campbell JP, et ali-ROP Research Consortium. Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity: performance of the “i-ROP” system and image features associated with expert diagnosis. Transl Vis Sci Technol. 2015;4(6):5. PMID:
- 16. Campbell JP, Ryan MC, Lore E, et alImaging & Informatics in Retinopathy of Prematurity Research Consortium. Diagnostic discrepancies in retinopathy of prematurity classification. Ophthalmology. 2016;123(8):1795–1801–. PMID:
- 17. Chiang MF, Jiang L, Gelman R, Du YE, Flynn JT. Interexpert agreement of plus disease diagnosis in retinopathy of prematurity. Arch Ophthalmol. 2007;125(7):875–880–. PMID:
- 18. Rao R, Jonsson NJ, Ventura C, et al. Plus disease in retinopathy of prematurity: diagnostic impact of field of view. Retina. 2012;32(6):1148–1155–. PMID:
- 19. Hewing NJ, Kaufman DR, Chan RV, Chiang MF. Plus disease in retinopathy of prematurity: qualitative analysis of diagnostic process by experts. JAMA Ophthalmol. 2013;131(8):1026–1032–. PMID:
- 20. Redd TK, Campbell JP, Brown JM, et alImaging and Informatics in Retinopathy of Prematurity (i-ROP) Research Consortium. Evaluation of a deep learning image assessment system for detecting severe retinopathy of prematurity. Br J Ophthalmol. Published online ahead of print November 23, 2018. PMID:
- 21. Campbell JP, Ataer-Cansizoglu E, Bolon-Canedo V, et alImaging and Informatics in ROP (i-ROP) Research Consortium. Expert diagnosis of plus disease in retinopathy of prematurity from computer-based image analysis. JAMA Ophthalmol. 2016;134(6):651–657–. PMID:
- 22. Kalpathy-Cramer J, Campbell JP, Erdogmus D, et alImaging and Informatics in Retinopathy of Prematurity Research Consortium. Plus disease in retinopathy of prematurity: improving diagnosis by ranking disease severity and using quantitative image analysis. Ophthalmology. 2016;123(11):2345–2351–. PMID:
- 23. Brown JM, Campbell JP, Beers A, et alImaging and Informatics in Retinopathy of Prematurity (i-ROP) Research Consortium. Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks. JAMA Ophthalmol. 2018;136(7):803–810–. PMID:
- 24. Taylor S, Brown JM, Gupta K, et alImaging and Informatics in Retinopathy of Prematurity Consortium. Monitoring disease progression with a quantitative severity scale for retinopathy of prematurity using deep learning. JAMA Ophthalmol. 2019;137(9):1022–1028–. PMID:
- 25. Gupta K, Campbell JP, Taylor S, et alImaging and Informatics in Retinopathy of Prematurity Consortium. A quantitative severity scale for retinopathy of prematurity using deep learning to monitor disease regression after treatment. JAMA Ophthalmol. 2019;137(9):1029–1036–. PMID:
- 26. Chen JS, Coyner AS, Ostmo S, et al. Deep learning for the diagnosis of stage in retinopathy of prematurity: accuracy and generalizability across populations and cameras. Ophthalmol Retina. 2021;5(10):1027–1035–. PMID:
- 27. Campbell JP, Singh P, Redd TK, et al. Applications of artificial intelligence for retinopathy of prematurity screening. Pediatrics. 2021;147(3):e2020016618. PMID:
- 28. Valikodath N, Cole E, Al-Khaled T, et al. Utility of an automated deep learning tool in a low-income country for retinopathy of prematurity. Invest Ophthalmol Vis Sci. 2020;61(7):1637–.
- 29. Al-Khaled T, Valikodath N, Cole E, et al. Evaluation of an artificial intelligence system (i-ROP DL) for retinopathy of prematurity screening in Nepal using the Forus 3nethra neo and in Mongolia using the Retcam Portable®. Invest Ophthalmol Vis Sci. 2021;62(8):3269–.
- 30. White F. Tech that detects cause of preemie blindness gets federal nod. Published January 30, 2020. Accessed July 29, 2020. https://news.ohsu.edu/2020/01/30/tech-that-detects-cause-of-preemie-blindness-gets-federal-nod
- 31. Al-Khaled T, Valikodath NG, Patel SN, et al. Addressing the third epidemic of retinopathy of prematurity through telemedicine and technology: a systematic review. J Pediatr Ophthalmol Strabismus. 2021;58(4):261–269–. PMID:
- 32. Cole E, Beca F, Gil GL, et al. iTeleGEN: development of a web-based diabetic retinopathy education module. Invest Ophthalmol Vis Sci. 2019;60(9):5307–.
- 33. Scanzera A, Cole E, Valikodath N, et al. Implementation of COVID-19 protocols and tele-triage in an academic ophthalmology department. J Acad Ophthalmol. 2020;12(2):151–158–.
- 34. Chiang MF, Quinn GE, Fielder AR, et al. International Classification of Retinopathy of Prematurity, Third Edition. Ophthalmology. 2021;128(10):e51–e68–. doi: