With the advances in artificial intelligence (AI), data‐driven algorithms are becoming increasingly popular in the medical domain. However, due to the nonlinear and complex behavior of many of these algorithms, decision‐making by such algorithms is not trustworthy for clinicians and is considered a black‐box process. Hence, the scientific community has introduced explainable artificial intelligence (XAI) to remedy the problem. This systematic scoping review investigates the application of XAI in breast cancer detection and risk prediction. We conducted a comprehensive search on Scopus, IEEE Explore, PubMed, and Google Scholar (first 50 citations) using a systematic search strategy. The search spanned from January 2017 to July 2023, focusing on peer‐reviewed studies implementing XAI methods in breast cancer datasets. Thirty studies met our inclusion criteria and were included in the analysis. The results revealed that SHapley Additive exPlanations (SHAP) is the top model‐agnostic XAI technique in breast cancer research in terms of usage, explaining the model prediction results, diagnosis and classification of biomarkers, and prognosis and survival analysis. Additionally, the SHAP model primarily explained tree‐based ensemble machine learning models. The most common reason is that SHAP is model agnostic, which makes it both popular and useful for explaining any model prediction. Additionally, it is relatively easy to implement effectively and completely suits performant models, such as tree‐based models. Explainable AI improves the transparency, interpretability, fairness, and trustworthiness of AI‐enabled health systems and medical devices and, ultimately, the quality of care and outcomes.
Zhang B, Vakanski A, Xian M. BI‐RADS‐NET‐V2: a composite multi‐task neural network for computer‐aided diagnosis of breast cancer in ultrasound images with semantic and quantitative explanations. IEEE Access. 2023;11:79480–94. https://doi.org/10.1109/ACCESS.2023.3298569
Hayes‐Roth F. Rule‐based systems. Commun ACM. 1985;28(9):921–32. https://doi.org/10.1145/4284.4286
van Giffen B, Herhausen D, Fahse T. Overcoming the pitfalls and perils of algorithms: a classification of machine learning biases and mitigation methods. J Bus Res. 2022;144:93–106. https://doi.org/10.1016/j.jbusres.2022.01.076
Shaban‐Nejad A, Michalowski M, Brownstein JS, Buckeridge DL. Guest editorial explainable AI: towards fairness, accountability, transparency and trust in healthcare. IEEE J Biomed Health Inform. 2021;25(7):2374–5. https://doi.org/10.1109/JBHI.2021.3088832
Quinn TP, Senadeera M, Jacobs S, Coghlan S, Le V. Trust and medical AI: the challenges we face and the expertise needed to overcome them. J Am Med Inform Asso. 2020;28(4):890–4. https://doi.org/10.1093/jamia/ocaa268
Pan SJ, Yang Q. A survey on transfer learning. IEEE Trans Knowl Data Eng. 2010;22(10):1345–59. https://doi.org/10.1109/TKDE.2009.191
Gunning D, Aha D. DARPA's explainable artificial intelligence (XAI) program. AI Magazine. 2019;40(2):44–58. https://doi.org/10.1609/aimag.v40i2.2850
Shaban‐Nejad A, Michalowski M, Buckeridge D. Explainable AI in healthcare and medicine: building a culture of transparency and accountability. Stud Comp Intel. 2020;914:344. https://doi.org/10.1007/978-3-030-53352-6
Luo W, Phung D, Tran T, Gupta S, Rana S, Karmakar C, et al. Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J Med Internet Res. 2016;18(12):e323. https://doi.org/10.2196/jmir.5870
Miller T. Explanation in artificial intelligence: insights from the social sciences. Artif Intel. 2019;267:1–38. https://doi.org/10.1016/j.artint.2018.07.007
Loh HW, Ooi CP, Seoni S, Barua PD, Molinari F, Acharya UR. Application of explainable artificial intelligence for healthcare: a systematic review of the last decade (2011–2022). Comp Methods Prog Biomed. 2022;226:107161. https://doi.org/10.1016/j.cmpb.2022.107161
Bharati S, Mondal MRH, Podder P. A review on explainable artificial intelligence for healthcare: why, how, and when? IEEE Trans Artif Intel. 2023;5:1–15. https://doi.org/10.1109/TAI.2023.3266418
Di Martino F, Delmastro F. Explainable AI for clinical and remote health applications: a survey on tabular and time series data. Artif Intel Rev. 2023;56(6):5261–315. https://doi.org/10.1007/s10462-022-10304-3
Hauser K, Kurz A, Haggenmüller S, Maron RC, von Kalle C, Utikal JS, et al. Explainable artificial intelligence in skin cancer recognition: a systematic review. Eur J Cancer. 2022;167:54–69. https://doi.org/10.1016/j.ejca.2022.02.025
James G, Witten D, Hastie T, Tibshirani R, et al. An introduction to statistical learning. vol. 112. New York, NY, USA: Springer; 2013.
Geurts P, Ernst D, Wehenkel L. Extremely randomized trees. Mach Learn. 2006;63:3–42. https://doi.org/10.1007/s10994-006-6226-1
Breiman L. Random forests. Mach Learn. 2001;45:5–32. https://doi.org/10.1023/A:1010933404324
Freund Y, Schapire RE. A decision‐theoretic generalization of on‐line learning and an application to boosting. J Comp Syst Sci. 1997;55(1):119–39. https://doi.org/10.1006/jcss.1997.1504
Sagi O, Rokach L. Ensemble learning: a survey. WIREs Data Min Knowl Disc. 2018;8(4):e1249. https://doi.org/10.1002/widm.1249
Shaban‐Nejad A, Michalowski M, Bianco S. Multimodal artificial intelligence: next wave of innovation in healthcare and medicine. Stud Comp Intel. 2022;1060:1–9.
Cybenko GV. Approximation by superpositions of a sigmoidal function. Math Control Signals Syst. 1989;2:303–14. https://doi.org/10.1007/BF02551274
Hornik K, Stinchcombe M, White H. Multilayer feedforward networks are universal approximators. Neural Netw. 1989;2(5):359–66. https://doi.org/10.1016/0893-6080(89)90020-8
Baydin AG, Pearlmutter BA, Radul AA, Siskind JM. Automatic differentiation in machine learning: a survey. J Mach Learn Res. 2018;18:1–43. https://doi.org/10.48550/arXiv.1502.05767
Lecun Y, Bottou L, Bengio Y, Haffner P. Gradient‐based learning applied to document recognition. Proc IEEE. 1998;86(11):2278–324. https://doi.org/10.1109/5.726791
Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Commun ACM. 2017;60(6):84–90. https://doi.org/10.1145/3065386
Szegedy C, Ioffe S, Vanhoucke V, Alemi A. Inception‐v4, inception‐ResNet and the impact of residual connections on learning. Proc AAAI Conf Artif Intel. 2017;31(1):4278–84. https://doi.org/10.1609/aaai.v31i1.11231
Huang G, Liu Z, Pleiss G, Weinberger KQ. Convolutional networks with dense connectivity. IEEE Trans Pattern Anal Mach Intel. 2022;44(12):8704–16. https://doi.org/10.1109/TPAMI.2019.2918284
Hochreiter S, Schmidhuber J. Long short‐term memory. Neural Comput. 1997;9(8):1735–80. https://doi.org/10.1162/neco.1997.9.8.1735
Schuster M, Paliwal KK. Bidirectional recurrent neural networks. IEEE Trans Signal Proces. 1997;45(11):2673–81. https://doi.org/10.1109/78.650093
van der Veer SN, Riste L, Cheraghi‐Sohi S, Phipps DL, Tully MP, Bozentko K, et al. Trading off accuracy and explainability in AI decision‐making: findings from 2 citizens' juries. J Am Med Inform Assoc. 2021;28(10):2128–38. https://doi.org/10.1093/jamia/ocab127
Pintelas E, Livieris IE, Pintelas P. A grey‐box ensemble model exploiting black‐box accuracy and white‐box intrinsic interpretability. Algorithms. 2020;13(1):1–17. https://doi.org/10.3390/a13010017
Bennetot A, Franchi G, Ser JD, Chatila R, Díaz‐Rodríguez N. Greybox XAI: a neural‐symbolic learning framework to produce interpretable predictions for image classification. Knowl Based Syst. 2022;258:109947. https://doi.org/10.1016/j.knosys.2022.109947
Wanner J, Herm LV, Heinrich K, Janiesch C, Zschech P. White, grey, black: effects of XAI augmentation on the confidence in ai‐based decision support systems. Forty‐First International Conference on Information Systems, India; 2020.
Bohlin TP. Practical grey‐box process identification: theory and applications. London, UK: Springer Science & Business Media; 2006.
Vilone G, Longo L. Classification of explainable artificial intelligence methods through their output formats. Mach Learn Knowl Extract. 2021;3(3):615–61. https://doi.org/10.3390/make3030032
Adadi A, Berrada M. Peeking inside the black‐box: a survey on explainable artificial intelligence (XAI). IEEE Access. 2018;6:52138–60. https://doi.org/10.1109/ACCESS.2018.2870052
Samek W, Montavon G, Lapuschkin S, Anders CJ, Müller KR. Explaining deep neural networks and beyond: a review of methods and applications. Proc IEEE. 2021;109(3):247–78. https://doi.org/10.1109/JPROC.2021.3060483
Clement T, Kemmerzell N, Abdelaal M, Amberg M. XAIR: a systematic metareview of explainable AI (XAI) aligned to the software development process. Mach Learn Knowl Extract. 2023;5(1):78–108. https://doi.org/10.3390/make5010006
Rumelhart DE, Hinton GE, Williams RJ. Learning internal representations by error propagation. San Diego, CA, USA: California University of San Diego La Jolla Institute for Cognitive Science; 1985.
LeCun Y. A learning scheme for asymmetric threshold networks. Proc Cognit. 1985;85(537):599–604.
Parker DB. Learning‐logic: casting the cortex of the human brain in silicon. Tech Rep. 1985;47.
Tober M. PubMed, ScienceDirect, Scopus or Google Scholar—which is the best search engine for an effective literature research in laser medicine? Med Laser Appl. 2011;26(3):139–44. https://doi.org/10.1016/j.mla.2011.05.006
Chakraborty D, Ivan C, Amero P, Khan M, Rodriguez‐Aguayo C, Başağaoğlu H, et al. Explainable artificial intelligence reveals novel insight into tumor microenvironment conditions linked with better prognosis in patients with breast cancer. Cancers. 2021;13(14):1–15. https://doi.org/10.3390/cancers13143450
Moncada‐Torres A, van Maaren MC, Hendriks MP, Siesling S, Geleijnse G. Explainable machine learning can outperform cox regression predictions and provide insights in breast cancer survival. Sci Rep. 2021;11. https://doi.org/10.1038/s41598-021-86327-7
Rezazadeh A, Jafarian Y, Kord A. Explainable ensemble machine learning for breast cancer diagnosis based on ultrasound image texture features. Forecasting. 2022;4(1):262–74. https://doi.org/10.3390/forecast4010015
Al‐Dhabyani W, Gomaa M, Khaled H, Fahmy A. Dataset of breast ultrasound images. Data Brief. 2020;28:104863. https://doi.org/10.1016/j.dib.2019.104863
Nahid AA, Raihan MJ, Bulbul AAM. Breast cancer classification along with feature prioritization using machine learning algorithms. Health Technol. 2022;12(6):1061–9. https://doi.org/10.1007/s12553-022-00710-6
Yu H, Chen F, Lam KO, Yang L, Wang Y, Jin JY, et al. Potential determinants for radiation‐induced lymphopenia in patients with breast cancer using interpretable machine learning approach. Front Immunol. 2022;13. https://doi.org/10.3389/fimmu.2022.768811
Meshoul S, Batouche A, Shaiba H, AlBinali S. Explainable multi‐class classification based on integrative feature selection for breast cancer subtyping. Mathematics. 2022;10(22). https://doi.org/10.3390/math10224271
Kumar S, Das A. Peripheral blood mononuclear cell derived biomarker detection using explainable artificial intelligence (XAI) provides better diagnosis of breast cancer. Comp Biol Chem. 2023;104:107867. https://doi.org/10.1016/j.compbiolchem.2023.107867
Silva‐Aravena F, NúñezDelafuente H, Gutiérrez‐Bahamondes JH, Morales J. A hybrid algorithm of ML and XAI to prevent breast cancer: a strategy to support decision making. Cancers. 2023;15(9):1–18. https://doi.org/10.3390/cancers15092443
Nindrea RD, Usman E, Katar Y, Darma IY, Warsiti, Hendriyani H, et al. Dataset of Indonesian women's reproductive, high‐fat diet and body mass index risk factors for breast cancer. Data in Brief. 2021;36:107107. https://doi.org/10.1016/j.dib.2021.107107
Massafra R, Fanizzi A, Amoroso N, Bove S, Comes MC, Pomarico D, et al. Analyzing breast cancer invasive disease event classification through explainable artificial intelligence. Front Med. 2023;10. https://doi.org/10.3389/fmed.2023.1116354
Vrdoljak J, Boban Z, Baric D, Segvic D, Kumric M, Avirovic M, et al. Applying explainable machine learning models for detection of breast cancer lymph node metastasis in patients eligible for neoadjuvant treatment. Cancers. 2023;15(3). https://doi.org/10.3390/cancers15030634
Mohi Uddin KM, Biswas N, Rikta ST, Dey SK, Qazi A. XML‐LightGBMDroid: a self‐driven interactive mobile application utilizing explainable machine learning for breast cancer diagnosis. Eng Rep. 2023;5:e12666. https://doi.org/10.1002/eng2.12666
Zhao X, Jiang C. The prediction of distant metastasis risk for male breast cancer patients based on an interpretable machine learning model. BMC Med Informat Decision Making. 2023;23(1):74. https://doi.org/10.1186/s12911-023-02166-8
Cordova C, Muñoz R, Olivares R, Minonzio JG, Lozano C, Gonzalez P, et al. HER2 classification in breast cancer cells: a new explainable machine learning application for immunohistochemistry. Oncol Lett. 2023;25(2):1–9. https://doi.org/10.3892/ol.2022.13630
Kaplun D, Krasichkov A, Chetyrbok P, Oleinikov N, Garg A, Pannu HS. Cancer cell profiling using image moments and neural networks with model agnostic explainability: a case study of breast cancer histopathological (BreakHis) database. Mathematics. 2021;9(20):1–20. https://doi.org/10.3390/math9202616
Spanhol FA, Oliveira LS, Petitjean C, Heutte L. A dataset for breast cancer histopathological image classification. IEEE Trans Biomed Eng. 2016;63(7):1455–62. https://doi.org/10.1109/TBME.2015.2496264
Saarela M, Jauhiainen S. Comparison of feature importance measures as explanations for classification models. SN Appl Sci. 2021;3. https://doi.org/10.1007/s42452-021-04148-9
Adnan N, Zand M, Huang THM, Ruan J. Construction and evaluation of robust interpretation models for breast cancer metastasis prediction. IEEE/ACM Trans Comp Biol Bioinform. 2022;19(3):1344–53. https://doi.org/10.1109/TCBB.2021.3120673
Staiger C, Cadot S, Györffy B, Wessels L, Klau G. Current composite‐feature classification methods do not outperform simple single‐genes classifiers in breast cancer prognosis. Front Genet. 2013;4. https://doi.org/10.3389/fgene.2013.00289
Maouche I, Terrissa LS, Benmohammed K, Zerhouni N. An explainable AI approach for breast cancer metastasis prediction based on clinicopathological data. IEEE Trans Biomed Eng. 2023;70:1–9. https://doi.org/10.1109/TBME.2023.3282840
Slaoui M, Mouh FZ, Ghanname I, Razine R, El Mzibri M, Amrani M. Outcome of breast cancer in Moroccan young women correlated to clinic‐pathological features, risk factors and treatment: a comparative study of 716 cases in a single institution. PLoS One. 2016;11(10):1–14. https://doi.org/10.1371/journal.pone.0164841
Deshmukh S, Behera BK, Mulay P, Ahmed EA, Al‐Kuwari S, Tiwari P, et al. Explainable quantum clustering method to model medical data. Knowl Based Syst. 2023;267:110413. https://doi.org/10.1016/j.knosys.2023.110413
Qi X, Zhang L, Chen Y, Pi Y, Chen Y, Lv Q, et al. Automated diagnosis of breast ultrasonography images using deep neural networks. Med Image Anal. 2019;52:185–98. https://doi.org/10.1016/j.media.2018.12.006
Zhou LQ, Wu XL, Huang SY, Wu GG, Ye HR, Wei Q, et al. Lymph node metastasis prediction from primary breast cancer US images using deep learning. Radiology. 2020;294(1):19–28. https://doi.org/10.1148/radiol.2019190372
Huang Z, Zhu X, Ding M, Zhang X. Medical image classification using a light‐weighted hybrid neural network based on PCANet and DenseNet. IEEE Access. 2020;8:24697–712. https://doi.org/10.1109/ACCESS.2020.2971225
Xi P, Guan H, Shu C, Borgeat L, Goubran R. An integrated approach for medical abnormality detection using deep patch convolutional neural networks. Vis Comput. 2020;36(9):1869–82. https://doi.org/10.1007/s00371-019-01775-7
Kim J, Kim HJ, Kim C, Lee JH, Kim KW, Park YM, et al. Weakly‐supervised deep learning for ultrasound diagnosis of breast cancer. Sci Rep. 2021;11:24382. https://doi.org/10.1038/s41598-021-03806-7
El Adoui M, Drisis S, Benjelloun M. Multi‐input deep learning architecture for predicting breast tumor response to chemotherapy using quantitative MR images. Int J Comp Assist Radiol Surg. 2020;15. https://doi.org/10.1007/s11548-020-02209-9
Hussain SM, Buongiorno D, Altini N, Berloco F, Prencipe B, Moschetta M, et al. Shape‐based breast lesion classification using digital tomosynthesis images: the role of explainable artificial intelligence. Appl Sci. 2022;12(12). https://doi.org/10.3390/app12126230
Bevilacqua V, Brunetti A, Guerriero A, Trotta GF, Telegrafo M, Moschetta M. A performance comparison between shallow and deeper neural networks supervised classification of tomosynthesis breast lesions images. Cogn Syst Res. 2019;53:3–19. https://doi.org/10.1016/j.cogsys.2018.04.011
Agbley BLY, Li JP, Haq AU, Bankas EK, Mawuli CB, Ahmad S, et al. Federated fusion of magnified histopathological images for breast tumor classification in the Internet of medical things. IEEE J Biomed Health Inform. 2023:1–12. https://doi.org/10.1109/JBHI.2023.3256974
Gerbasi A, Clementi G, Corsi F, Albasini S, Malovini A, Quaglini S, et al. DeepMiCa: automatic segmentation and classification of breast MIcroCAlcifications from mammograms. Comp Methods Prog Biomed. 2023;235:107483. https://doi.org/10.1016/j.cmpb.2023.107483
Moreira IC, Amaral I, Domingues I, Cardoso A, Cardoso MJ, Cardoso JS. INbreast: toward a full‐field digital mammographic database. Acad Radiol. 2012;19(2):236–48. https://doi.org/10.1016/j.acra.2011.09.014
To T, Lu T, Jorns JM, Patton M, Schmidt TG, Yen T, et al. Deep learning classification of deep ultraviolet fluorescence images toward intra‐operative margin assessment in breast cancer. Front Oncol. 2023;13. https://doi.org/10.3389/fonc.2023.1179025
Lu T, Jorns JM, Patton M, Fisher R, Emmrich A, Doehring T, et al. Rapid assessment of breast tumor margins using deep ultraviolet fluorescence scanning microscopy. J Biomed Opt. 2020;25(12):126501. https://doi.org/10.1117/1.JBO.25.12.126501
Grisci BI, Krause MJ, Dorn M. Relevance aggregation for neural networks interpretability and knowledge discovery on tabular data. Inform Sci. 2021;559:111–29. https://doi.org/10.1016/j.ins.2021.01.052
Feltes BC, Chandelier EB, Grisci BI, Dorn M. CuMiDa: an extensively curated microarray database for benchmarking and testing of machine learning approaches in cancer research. J Comp Biol. 2019;26(4):376–86. https://doi.org/10.1089/cmb.2018.0238
Chereda H, Bleckmann A, Menck K, Perera‐Bel J, Stegmaier P, Auer F, et al. Explaining decisions of graph convolutional neural networks: patient‐specific molecular subnetworks responsible for metastasis prediction in breast cancer. Genome Med. 2021;13. https://doi.org/10.1186/s13073-021-00845-7
Barrett T, Wilhite SE, Ledoux P, Evangelista C, Kim IF, Tomashevsky M, et al. NCBI GEO: archive for functional genomics data sets—update. Nucl Acids Res. 2012;41(D1):D991–5. https://doi.org/10.1093/nar/gks1193
Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad‐CAM: visual explanations from deep networks via gradient‐based localization. Int J Comput Vision. 2020;128(2):336–59. https://doi.org/10.1007/s11263-019-01228-7
Bach S, Binder A, Montavon G, Klauschen F, Müller KR, Samek W. On pixel‐wise explanations for non‐linear classifier decisions by layer‐wise relevance propagation. PLoS One. 2015;10(7):1–46. https://doi.org/10.1371/journal.pone.0130140
Brenas JH, Shaban‐Nejad A. Health intervention evaluation using semantic explainability and causal reasoning. IEEE Access. 2020;8:9942–52. https://doi.org/10.1109/ACCESS.2020.2964802
Brakefield WS, Ammar N, Shaban‐Nejad A. An urban population health observatory for disease causal pathway analysis and decision support: underlying explainable artificial intelligence model. JMIR Form Res. 2022;6(7):e36055. https://doi.org/10.2196/36055
Ammar N, Shaban‐Nejad A. Explainable artificial intelligence recommendation system by leveraging the semantics of adverse childhood experiences: proof‐of‐concept prototype development. JMIR Med Inform. 2020;8(11):e18752. https://doi.org/10.2196/18752
Chanda T, Hauser K, Hobelsberger S, Bucher TC, Garcia CN, Wies C, et al. Dermatologist‐like explainable AI enhances trust and confidence in diagnosing melanoma. Nat Commun. 2024;15(1):524. https://doi.org/10.1038/s41467-023-43095-4
Borole P, Rajan A. Building trust in deep learning‐based immune response predictors with interpretable explanations. Commun Biol. 2024;7(1):279. https://doi.org/10.1038/s42003-024-05968-2
Fania A, Monaco A, Amoroso N, Bellantuono L, Cazzolla Gatti R, Firza N, et al. Machine learning and XAI approaches highlight the strong connection between O3 and NO2 pollutants and Alzheimer's disease. Sci Rep. 2024;14(1):5385. https://doi.org/10.1038/s41598-024-55439-1
Ng MY, Youssef A, Miner AS, Sarellano D, Long J, Larson DB, et al. Perceptions of data set experts on important characteristics of health data sets ready for machine learning: a qualitative study. JAMA Netw Open. 2023;6(12):e2345892. https://doi.org/10.1001/jamanetworkopen.2023.45892
Friedman JH. Greedy function approximation: a gradient boosting machine. Ann Statist. 2001;29(5):1189–232. https://doi.org/10.1214/aos/1013203451
Wachter S, Mittelstadt B, Russell C. Counterfactual explanations without opening the black box: automated decisions and the GDPR. Harv JL Tech. 2017;31:841. https://doi.org/10.48550/arXiv.1711.00399
Montavon G, Lapuschkin S, Binder A, Samek W, Müller KR. Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recogn. 2017;65:211–22. https://doi.org/10.1016/j.patcog.2016.11.008
Erhan D, Bengio Y, Courville A, Vincent P. Visualizing higher‐layer features of a deep network. Univ Montreal. 2009;1341(3):1.
Huang Q, Yamada M, Tian Y, Singh D, Chang Y. GraphLIME: local interpretable model explanations for graph neural networks. IEEE Trans Knowl Data Eng. 2023;35(7):6968–72. https://doi.org/10.1109/TKDE.2022.3187455