Introduction: Our aim in this study was to summarize information on the use of intelligent models for predicting and diagnosing the Coronavirus disease 2019 (COVID-19) to help early and timely diagnosis of the disease.
Material and Methods: A systematic literature search included articles published until 20 April 2020 in PubMed, Web of Science, IEEE, ProQuest, Scopus, bioRxiv, and medRxiv databases. The search strategy consisted of two groups of keywords: A) Novel coronavirus, B) Machine learning. Two reviewers independently assessed original papers to determine eligibility for inclusion in this review. Studies were critically reviewed for risk of bias using prediction model risk of bias assessment tool.
Results: We gathered 1650 articles through database searches. After the full-text assessment 31 articles were included. Neural networks and deep neural network variants were the most popular machine learning type. Of the five models that authors claimed were externally validated, we considered external validation only for four of them. Area under the curve (AUC) in internal validation of prognostic models varied from .94 to .97. AUC in diagnostic models varied from 0.84 to 0.99, and AUC in external validation of diagnostic models varied from 0.73 to 0.94. Our analysis finds all but two studies have a high risk of bias due to various reasons like a low number of participants and lack of external validation.Conclusion: Diagnostic and prognostic models for COVID-19 show good to excellent discriminative performance. However, these models are at high risk of bias because of various reasons like a low number of participants and lack of external validation. Future studies should address these concerns. Sharing data and experiences for the development, validation, and updating of COVID-19 related prediction models is needed.
World Health Organization. Coronavirus disease (COVID-2019) situation reports 2020 [Internet]. 2020 [cited: 27 Jul 2021]. Available from: https://www.who.int/emergencies/diseases/novel-coronavirus-2019
Sohrabi C, Alsafi Z, O'Neill N, Khan M, Kerwan A, Al-Jabir A, et al. World Health Organization declares global emergency: A review of the 2019 novel coronavirus (COVID-19). Int J Surg. 2020; 76: 71-6. PMID: 32112977 DOI: 10.1016/j.ijsu.2020.02.034
World Health Organization. Coronavirus disease (COVID-19) pandemic 2020 [Internet]. 2020 [cited: 27 Jul 2021]. Available from: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/events-as-they-happen
Arabi YM, Murthy S, Webb S. COVID-19: A novel coronavirus and a novel challenge for critical care. Intensive Care Med. 2020; 46(5): 833-6. PMID: 32125458 DOI: 10.1007/s00134-020-05955-1
Grasselli G, Pesenti A, Cecconi M. Critical care utilization for the COVID-19 outbreak in Lombardy, Italy: Early experience and forecast during an emergency response. JAMA. 2020; 323(16): 1545-6. PMID: 32167538 DOI: 10.1001/jama.2020.4031
Xie J, Tong Z, Guan X, Du B, Qiu H, Slutsky AS. Critical care crisis and some recommendations during the COVID-19 epidemic in China. Intensive Care Med. 2020; 46(5): 837-40. PMID: 32123994 DOI: 10.1007/s00134-020-05979-7
Chung M, Bernheim A, Mei X, Zhang N, Huang M, Zeng X, et al. CT imaging features of 2019 novel coronavirus (2019-nCoV). Radiology. 2020; 295(1): 202-7. PMID: 32017661 DOI: 10.1148/radiol.2020200230
Yang X, Yu Y, Xu J, Shu H, Liu H, Wu Y, et al. Clinical course and outcomes of critically ill patients with SARS-CoV-2 pneumonia in Wuhan, China: A single-centered, retrospective, observational study. Lancet Respir Med. 2020; 8(5): 475-81. PMID: 32105632 DOI: 10.1016/S2213-2600(20)30079-5
Xia C, Li X, Wang X, Kong B, Chen Y, Yin Y, et al. A multi-modality network for cardiomyopathy death risk prediction with CMR images and clinical information. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer; 2019.
Kong B, Wang X, Bai J, Lu Y, Gao F, Cao K, et al. Learning tree-structured representation for 3D coronary artery segmentation. Comput Med Imaging Graph. 2020; 80: 101688. PMID: 31926366 DOI: 10.1016/j.compmedimag.2019.101688
Zaharchuk G, Gong E, Wintermark M, Rubin D, Langlotz C. Deep learning in neuroradiology. AJNR Am J Neuroradiol. 2018; 39(10): 1776-84. PMID: 29419402 DOI: 10.3174/ajnr.A5543
Yuan J, Liao H, Luo R, Luo J. Automatic radiology report generation based on multi-view image fusion and medical concept enrichment. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer; 2019.
Kermany DS, Goldbaum M, Cai W, Valentim CC, Liang H, Baxter SL, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell. 2018; 172(5): 1122-31. PMID: 29474911 DOI: 10.1016/j.cell.2018.02.010
Rajaraman S, Candemir S, Kim I, Thoma G, Antani S. Visualization and interpretation of convolutional neural network predictions in detecting pneumonia in pediatric chest radiographs. Appl Sci (Basel). 2018; 8(10): 1715. PMID: 32457819 DOI: 10.3390/app8101715
Depeursinge A, Chin AS, Leung AN, Terrone D, Bristow M, Rosen G, et al. Automated classification of usual interstitial pneumonia using regional volumetric texture analysis in high-resolution CT. Invest Radiol. 2015; 50(4): 261-7. PMID: 25551822 DOI: 10.1097/RLI.0000000000000127
Anthimopoulos M, Christodoulidis S, Ebner L, Christe A, Mougiakakou S. Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans Med Imaging. 2016; 35(5): 1207-16. PMID: 26955021 DOI: 10.1109/TMI.2016.2535865
Pal R, Sekh AA, Kar S, Prasad DK. Neural network based country wise risk prediction of COVID-19. Applied Sciences. 2020; 10(18): 6448.
Carr D. Sharing research data and findings relevant to the novel coronavirus (COVID-19) outbreak [Internet]. 2020 [cited: 17 Jul 2021]. Available from: https://wellcome.org/press-release/sharing-research-data-and-findings-relevant-novel-coronavirus-ncov-outbreak
Stewart LA, Clarke M, Rovers M, Riley RD, Simmonds M, Stewart G, et al. Preferred reporting items for a systematic review and meta-analysis of individual participant data: The PRISMA-IPD statement. JAMA. 2015; 313(16): 1657-65. PMID: 25919529 DOI: 10.1001/jama.2015.3656
Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan: A web and mobile app for systematic reviews. Syst Rev. 2016; 5(1): 210. PMID: 27919275 DOI: 10.1186/s13643-016-0384-4
Moons KG, de Groot JA, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: The CHARMS checklist. PLoS Med. 2014; 11(10): e1001744. PMID: 25314315 DOI: 10.1371/journal.pmed.1001744
Moons KG, Wolff RF, Riley RD, Whiting PF, Westwood M, Collins GS, et al. PROBAST: A tool to assess risk of bias and applicability of prediction model studies: Explanation and elaboration. Annals of Internal Medicine. 2019; 170: W1-33.
Wynants L, van Calster B, Bonten MM, Collins GS, Debray TP, de Vos M, et al. Prediction models for diagnosis and prognosis of covid-19 infection: Systematic review and critical appraisal. BMJ. 2020; 369: m1328. PMID: 32265220 DOI: 10.1136/bmj.m1328
Diaz-Quijano FA, da Silva JMN, Ganem F, Oliveira S, Vesga-Varela AL, Croda J. A model to predict SARS-CoV-2 infection based on the first three-month surveillance data in Brazil. Trop Med Int Health. 2020; 25(11): 1385-94. PMID: 32790891 DOI: 10.1111/tmi.13476
de Moraes Batista AF, Miraglia JL, Donato THR, Chiavegatto Filho ADP. COVID-19 diagnosis prediction in emergency care patients: A machine learning approach [Internet]. 2020 [cited: 14 Apr 2020]. Available from: http://medrxiv.org/lookup/doi/10.1101/2020.04.04.20052092
Ozturk S, Ozkaya U, Barstugan M. Classification of coronavirus images using Shrunken features. Int J Imaging Syst Technol. 2020; Online ahead of print. PMID: 32904960 DOI: 10.1002/ima.22469
Mahdy LN, Hassanien AE, Ezzat KA, Elmousalami HH, Aboul Ella H. Automatic X-ray COVID-19 lung image classification system based on multi-level thresholding and support vector machine [Internet]. 2020 [cited: 14 Apr 2020]. Available from: https://www.medrxiv.org/content/10.1101/2020.03.30.20047787v1
Bukhari SUK, Bukhari SSK, Syed A, Shah SSH. The diagnostic evaluation of convolutional neural network (CNN) for the assessment of chest X-ray of patients infected with COVID-19 [Internet]. 2020 [cited: 14 Apr 2020]. Available from: http://medrxiv.org/lookup/doi/10.1101/2020.03.26.20044610
Fu M, Yi S-L, Zeng Y, Ye F, Li Y, Dong X, et al. Deep learning-based recognizing COVID-19 and other common infectious diseases of the lung by chest CT scan images [Internet]. 2020 [cited: 14 Apr 2020]. Available from: http://medrxiv.org/lookup/doi/10.1101/2020.03.28.20046045
Zhou M, Chen Y, Wang D, Xu Y, Yao W, Huang J, et al. Improved deep learning model for differentiating novel coronavirus pneumonia and influenza pneumonia. Ann Transl Med. 2021; 9(2): 111. PMID: 33569413 DOI: 10.21037/atm-20-5328
Castiglioni I, Ippolito D, Interlenghi M, Monti CB, Salvatore C, Schiaffino S, et al. Artificial intelligence applied on chest X-ray can aid in the diagnosis of COVID-19 infection: A first experience from Lombardy, Italy. Eur Radiol Exp. 2021; 5(1): 7. PMID: 33527198 DOI: 10.1186/s41747-020-00203-z
Wang S, Kang B, Ma J, Zeng X, Xiao M, Guo J, et al. A deep learning algorithm using CT images to screen for corona virus disease (COVID-19). Eur Radiol. 2021; 31(8): 6096-104. PMID: 33629156 DOI: 10.1007/s00330-021-07715-1
Song Y, Zheng S, Li L, Zhang X, Zhang X, Huang Z, et al. Deep learning enables accurate diagnosis of novel coronavirus (COVID-19) with CT images. IEEE/ACM Trans Comput Biol Bioinform. 2021; Online ahead of print. PMID: 33705321 DOI: 10.1109/TCBB.2021.3065361
Zheng C, Deng X, Fu Q, Zhou Q, Feng J, Ma H, et al. A weakly-supervised framework for COVID-19 classification and lesion localization from chest CT. IEEE Transactions on Medical Imaging. 2020; 39(8): 2615-25.
Chen J, Wu L, Zhang J, Zhang L, Gong D, Zhao Y, et al. Deep learning-based model for detecting 2019 novel coronavirus pneumonia on high-resolution computed tomography: A prospective study. Sci Rep. 2020; 10(1): 19196. PMID: 33154542 DOI: 10.1038/s41598-020-76282-0
Meng Z, Wang M, Song H, Guo S, Zhou Y, Li W, et al. Development and utilization of an intelligent application for aiding COVID-19 diagnosis [Internet]. 2020 [cited: 14 Apr 2020]. Available from: http://medrxiv.org/lookup/doi/10.1101/2020.03.18.20035816
Feng C, Huang Z, Wang L, Chen X, Zhai Y, Zhu F, et al. A novel triage tool of artificial intelligence assisted diagnosis aid system for suspected COVID-19 pneumonia in fever clinics. Ann Transl Med. 2021; 9(3): 201. PMID: 33708828 DOI: 10.21037/atm-20-3073
Jin S, Wang B, Xu H, Luo C, Wei L, Zhao W, et al. AI-assisted CT imaging analysis for COVID-19 screening: Building and deploying a medical AI system in four weeks. Appl Soft Comput. 2021; 98: 106897. PMID: 33199977 DOI: 10.1016/j.asoc.2020.106897
Li L, Qin L, Xu Z, Yin Y, Wang X, Kong B, et al. Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT. Radiology. 2020; 200905. PMID: 32191588 DOI: 10.1148/radiol.2020200905
Maghdid HS, Asaad AT, Ghafoor KZ, Sadiq AS, Khan MK. Diagnosing COVID-19 pneumonia from X-ray and CT images using deep learning and transfer learning algorithms [Internet]. 2020 [cited: 15 Apr 2020]. Available from: http://arxiv.org/abs/2004.00038
Farooq M, Hafeez A. COVID-ResNet: A deep learning framework for screening of COVID19 from radiographs [Internet]. 2020 [cited: 15 Apr 2020]. Available from: http://arxiv.org/abs/2003.14395
Wang L, Wong A. COVID-Net: A tailored deep convolutional neural network design for detection of COVID-19 cases from chest radiography images. Scientific Reports. 2020; 10: 19549.
Zhang J, Xie Y, Li Y, Shen C, Xia Y. COVID-19 screening on chest X-ray images using deep learning based anomaly detection [Internet]. 2020 [cited: 15 Apr 2020]. Available from: http://arxiv.org/abs/2003.12338
Chowdhury MEH, Rahman T, Khandakar A, Mazhar R, Kadir MA, Mahbub ZB, et al. Can AI help in screening viral and COVID-19 pneumonia? IEEE Access. 2020; 8: 132665-76.
Apostolopoulos ID, Bessiana T. Covid-19: Automatic detection from X-Ray images utilizing transfer learning with convolutional neural networks. Phys Eng Sci Med. 2020; 43(2): 635-40. PMID: 32524445 DOI: 10.1007/s13246-020-00865-4
Hemdan EE-D, Shouman MA, Karar ME. COVIDX-Net: A framework of deep learning classifiers to diagnose COVID-19 in X-ray images [Internet]. 2020 [cited: 15 Apr 2020]. Available from: http://arxiv.org/abs/2003.11055
Narin A, Kaya C, Pamuk Z. Automatic detection of coronavirus disease (COVID-19) using X-ray images and deep convolutional neural networks. Pattern Anal Appl. 2021; Online ahead of print. PMID: 33994847 DOI: 10.1007/s10044-021-00984-y
Barstugan M, Ozkaya U, Ozturk S. Coronavirus (COVID-19) classification using CT images by machine learning methods [Internet]. 2020 [cited: 15 Apr 2020]. Available from: http://arxiv.org/abs/2003.09424
Abbas A, Abdelsamea M, Gaber M. Classification of COVID-19 in chest X-ray images using DeTraC deep convolutional neural network. Appl Intell. 2021; 51: 854–64.
Shi F, Xia L, Shan F, Wu D, Wei Y, Yuan H, et al. Large-scale screening of COVID-19 from community acquired pneumonia using infection size-aware classification. Phys Med Biol. 2021; Online ahead of print. PMID: 33607630 DOI: 10.1088/1361-6560/abe838
Xu X, Jiang X, Ma C, Du P, Li X, Lv S, et al. A deep learning system to screen coronavirus disease 2019 pneumonia. Engineering (Beijing). 2020; 6(10): 1122-9. PMID: 32837749 DOI: 10.1016/j.eng.2020.04.010
Pourhomayoun M, Shakibi M. Predicting mortality risk in patients with COVID-19 using artificial intelligence to help medical decision-making. Smart Health (Amst). 2021; 20: 100178. PMID: 33521226 DOI: 10.1016/j.smhl.2020.100178
Sarkar J, Chakrabarti P. A machine learning model reveals older age and delayed hospitalization as predictors of mortality in patients with COVID-19 [Internet]. 2020 [cited: 14 Apr 2020]. Available from: http://medrxiv.org/lookup/doi/10.1101/2020.03.25.20043331
Fang C, Bai X, Zhou Y, Bai S, Liu Z, Chen Q, et al. Deep learning for predicting COVID-19 malignant progression. Medical Image Analysis. 2021; 72: 102096.
Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement. BMJ. 2015; 350: g7594. PMID: 25569120 DOI: 10.1136/bmj.g7594
van Calster B, McLernon D, van Smeden M, Wynants L, Steyerberg E. Calibration: The achilles heel of predictive analytics. BMC Med. 2019; 17(1): 230. PMID: 31842878 DOI: 10.1186/s12916-019-1466-7
Steyerberg EW. Clinical prediction models: Practical approach to development, validation, and updating. SpringerLink; 2019.
Steyerberg EW. Evaluation of clinical usefulness. In: Steyerberg EW. Clinical rediction models: Practical approach to development, validation, and updating. SpringerLink; 2019.
Riley RD, Ensor J, Snell KI, Debray TP, Altman DG, Moons KG, et al. External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: Opportunities and challenges. BMJ. 2016; 353: i3140. PMID: 27334381 DOI: 10.1136/bmj.i3140
Wynants L, Kent D, Timmerman D, Lundquist C, van Calster B. Untapped potential of multicenter studies: a review of cardiovascular risk prediction models revealed inappropriate analyses and wide variation in reporting. Diagn Progn Res. 2019; 3: 6. PMID: 31093576 DOI: 10.1186/s41512-019-0046-9
Cui Z, Gong G. The effect of machine learning regression algorithms and sample size on individualized behavioral prediction with functional connectivity features. Neuroimage. 2018; 178: 622-37. PMID: 29870817 DOI: 10.1016/j.neuroimage.2018.06.001
Chu C, Hsu A-L, Chou K-H, Bandettini P, Lin C, Alzheimer's Disease Neuroimaging Initiative. Does feature selection improve classification accuracy? Impact of sample size and feature selection on classification using anatomical magnetic resonance images. Neuroimage. 2012; 60(1): 59-70. PMID: 22166797 DOI: 10.1016/j.neuroimage.2011.11.066
Arbabshirani MR, Plis S, Sui J, Calhoun VD. Single subject prediction of brain disorders in neuroimaging: Promises and pitfalls. Neuroimage. 2017; 145: 137-65. PMID: 27012503 DOI: 10.1016/j.neuroimage.2016.02.079
Riley RD, Ensor J, Snell KI, Harrell FE, Martin GP, Reitsma JB, et al. Calculating the sample size required for developing a clinical prediction model. BMJ. 2020; 368: m441. PMID: 32188600 DOI: 10.1136/bmj.m441
Huang J, Ling CX. Using AUC and accuracy in evaluating learning algorithms. IEEE Transactions on Knowledge and Data Engineering. 2005;17(3): 299-310.
Jeni LA, Cohn JF, de la Torre F. Facing imbalanced data: Recommendations for the use of performance metrics. Humaine Association Conference on Affective Computing and Intelligent Interaction. IEEE; 2013.