Home Big Data Mining and Analytics Article
PDF (3 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript
Open Access

Segmentation-Guided Deep Learning for Glioma Survival Risk Prediction with Multimodal MRI

Jianhong Cheng1, Hulin Kuang2(), Songhan Yang1, Hailin Yue2, Jin Liu2, Jianxin Wang2()
Institute of Guizhou Aerospace Measuring and Testing Technology, Guiyang 550009, China
Hunan Provincial Key Laboratory on Bioinformatics, School of Computer Science and Engineering, Central South University, Changsha 410083, China
Show Author Information

Abstract

Glioma survival risk prediction is of great significance for the individualized treatment and assessment programs. Currently, most deep learning based survival prediction paradigms rely on invasive and expensive histopathology and genomics methods. However, magnetic resonance imaging (MRI) has emerged as a promising non-invasive alternative with significant prognostic potential. To leverage the benefits of MRI, we propose a segmentation-guided fully automated multimodal MRI-based survival network (SGS-Net), which can simultaneously perform glioma segmentation and survival risk prediction. Specifically, the task interrelation is addressed using a hybrid convolutional neural network-Transformer (CNN-Transformer) encoder to represent the shared high-level semantic features by co-training a decoder for glioma segmentation and a Cox model for survival prediction. Then, to ensure the effective representation of the high-level features, glioma segmentation as an auxiliary task is utilized to guide survival prediction by jointly optimizing the segmentation loss and the Cox partial log-likelihood loss. Furthermore, a pair-wise ranking loss is designed to allow the network to learn the survival difference between patients. To balance the multi-task losses, an uncertain weight manner is adopted to adaptively adjust the weights for preventing task bias. Finally, the proposed SGS-Net is assessed using a publicly available multi-institutional dataset. Experimental and visual results show that SGS-Net achieves promising segmentation performance and obtains a C-index of 81.07% for survival risk prediction, which outperforms several existing state-of-the-art methods and even histopathology-based methods. In addition, Kaplan-Meier survival analysis confirms that the prognosis risk generated by SGS-Net is consistent with the prior prognosis based on the grading or genotyping paradigms.

Keywords

deep learningglioma segmentationsurvival riskmultimodal magnetic resonance imaging (MRI)

References

[1]

M. Weller, W. Wick, and G. Reifenberger, Glioma, Nature Reviews Disease Primers, vol. 1, no. 1, p. 15040, 2015.

Crossref Google Scholar
[2]

F. Bray, J. Ferlay, I. Soerjomataram, R. L. Siegel, L. A. Torre, and A. Jemal, Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries, CA A Cancer J. Clin., vol. 68, no. 6, pp. 394–424, 2018.

Crossref Google Scholar
[3]
Q. T. Ostrom, N. Patil, G. Cioffi, K. Waite, C. Kruchko, and J. S. Barnholtz-Sloan, CBTRUS statistical report: Primary brain and other central nervous system tumors diagnosed in the United States in 2013–2017, Neuro-Oncology, vol. 22, no. Suppl 2, pp. iv1–iv96, 2020.
[4]

W. A. Flavahan, Y. Drier, B. B. Liau, S. M. Gillespie, A. S. Venteicher, A. O. Stemmer-Rachamimov, M. L. Suvà, and B. E. Bernstein, Insulator dysfunction and oncogene activation in IDH mutant gliomas, Nature, vol. 529, pp. 110–114, 2016.

Crossref Google Scholar
[5]

M. C. Chamberlain and D. Born, Prognostic significance of relative 1p/19q codeletion in oligodendroglial tumors, J. Neuro Oncol., vol. 125, no. 2, pp. 249–251, 2015.

Crossref Google Scholar
[6]

S. Haase, M. B. Garcia-Fabiani, S. Carney, D. Altshuler, F. J. Núñez, F. M. Méndez, F. Núñez, P. R. Lowenstein, and M. G. Castro, Mutant ATRX: Uncovering a new therapeutic target for glioma, Expert Opin. Ther. Targets, vol. 22, no. 7, pp. 599–613, 2018.

Crossref Google Scholar
[7]

N. Thon, S. Kreth, and F. W. Kreth, Personalized treatment strategies in glioblastoma: MGMT promoter methylation status, Onco. Targets Ther., vol. 6, pp. 1363–1372, 2013.

Google Scholar
[8]

J. Yan, Q. Sun, X. Tan, C. Liang, H. Bai, W. Duan, T. Mu, Y. Guo, Y. Qiu, W. Wang, et al., Image-based deep learning identifies glioblastoma risk groups with genomic and transcriptomic heterogeneity: A multi-center study, Eur. Radiol., vol. 33, no. 2, pp. 904–914, 2023.

Google Scholar
[9]

K. M. Boehm, P. Khosravi, R. Vanguri, J. Gao, and S. P. Shah, Harnessing multimodal data integration to advance precision oncology, Nat. Rev. Cancer, vol. 22, no. 2, pp. 114–126, 2022.

Crossref Google Scholar
[10]

J. C. Denny and F. S. Collins, Precision medicine in 2030—Seven ways to transform healthcare, Cell, vol. 184, no. 6, pp. 1415–1419, 2021.

Crossref Google Scholar
[11]

D. J. Leao, P. G. Craig, L. F. Godoy, C. C. Leite, and B. Policeni, Response assessment in neuro-oncology criteria for gliomas: Practical approach using conventional and advanced techniques, Am. J. Neuroradiol., vol. 41, no. 1, pp. 10–20, 2020.

Crossref Google Scholar
[12]
N. Cohen and A. Shashua, Inductive bias of deep convolutional networks through pooling geometry, presented at 5th International Conference on Learning Representations (ICLR), Toulon, France, 2017.
[13]

J. Chen, J. Mei, X. Li, Y. Lu, Q. Yu, Q. Wei, X. Luo, Y. Xie, E. Adeli, Y. Wang, et al., TransUNet: Rethinking the U-Net architecture design for medical image segmentation through the lens of transformers, Med. Image Anal., vol. 97, p. 103280, 2024.

Crossref Google Scholar
[14]

J. Cheng, J. Liu, H. Kuang, and J. Wang, A fully automated multimodal MRI-based multi-task learning for glioma segmentation and IDH genotyping, IEEE Trans. Med. Imag., vol. 41, no. 6, pp. 1520–1532, 2022.

Crossref Google Scholar
[15]

H. Kuang, Y. Wang, J. Liu, J. Wang, Q. Cao, B. Hu, W. Qiu, and J. Wang, Hybrid CNN-transformer network with circular feature interaction for acute ischemic stroke lesion segmentation on non-contrast CT scans, IEEE Trans. Med. Imag., vol. 43, no. 6, pp. 2303–2316, 2024.

Crossref Google Scholar
[16]

Y. Fujioka, N. Hata, Y. Akagi, D. Kuga, R. Hatae, Y. Sangatsuda, Y. Michiwaki, T. Amemiya, K. Takigawa, Y. Funakoshi, et al., Molecular diagnosis of diffuse glioma using a chip-based digital PCR system to analyze IDH, TERT, and H3 mutations in the cerebrospinal fluid, J. Neurooncol, vol. 152, no. 1, pp. 47–54, 2021.

Crossref Google Scholar
[17]

L. Di, D. G. Eichberg, K. Huang, A. H. Shah, A. M. Jamshidi, E. M. Luther, V. M. Lu, R. J. Komotar, M. E. Ivan, and S. H. Gultekin, Stimulated Raman histology for rapid intraoperative diagnosis of gliomas, World Neurosurg., vol. 150, pp. e135–e143, 2021.

Crossref Google Scholar
[18]

J. Yan, B. Zhang, S. Zhang, J. Cheng, X. Liu, W. Wang, Y. Dong, L. Zhang, X. Mo, Q. Chen, et al., Quantitative MRI-based radiomics for noninvasively predicting molecular subtypes and survival in glioma patients, NPJ Precis. Oncol., vol. 5, no. 1, p. 72, 2021.

Crossref Google Scholar
[19]

S. Gore, T. Chougule, J. Jagtap, J. Saini, and M. Ingalhalikar, A review of radiomics and deep predictive modeling in glioma characterization, Acad. Radiol., vol. 28, no. 11, pp. 1599–1621, 2021.

Crossref Google Scholar
[20]

P. Lambin, R. T. H. Leijenaar, T. M. Deist, J. Peerlings, E. E. C. de Jong, J. van Timmeren, S. Sanduleanu, R. T. H. M. Larue, A. J. G. Even, A. Jochems, et al., Radiomics: The bridge between medical imaging and personalized medicine, Nat. Rev. Clin. Oncol., vol. 14, no. 12, pp. 749–762, 2017.

Crossref Google Scholar
[21]

G. Singh, S. Manjila, N. Sakla, A. True, A. H. Wardeh, N. Beig, A. Vaysberg, J. Matthews, P. Prasanna, and V. Spektor, Radiomics and radiogenomics in gliomas: A contemporary update, Br. J. Cancer, vol. 125, no. 5, pp. 641–657, 2021.

Crossref Google Scholar
[22]

A. Traverso, L. Wee, A. Dekker, and R. Gillies, Repeatability and reproducibility of radiomic features: A systematic review, Int. J. Radiat. Oncol., vol. 102, no. 4, pp. 1143–1158, 2018.

Crossref Google Scholar
[23]

P. Mobadersany, S. Yousefi, M. Amgad, D. A. Gutman, J. S. Barnholtz-Sloan, J. E. Velázquez Vega, D. J. Brat, and L. A. D. Cooper, Predicting cancer outcomes from histology and genomics using convolutional networks, Proc. Natl. Acad. Sci. U.S.A., vol. 115, no. 13, pp. E2970–E2979, 2018.

Google Scholar
[24]

A. Cheerla and O. Gevaert, Deep learning with multimodal representation for pancancer prognosis prediction, Bioinform. Oxf. Engl., vol. 35, no. 14, pp. i446–i454, 2019.

Google Scholar
[25]

R. J. Chen, M. Y. Lu, J. Wang, D. F. K. Williamson, S. J. Rodig, N. I. Lindeman, and F. Mahmood, Pathomic fusion: An integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis, IEEE Trans. Med. Imag., vol. 41, no. 4, pp. 757–770, 2022.

Crossref Google Scholar
[26]

Y. S. Choi, S. S. Ahn, J. H. Chang, S. G. Kang, E. H. Kim, S. H. Kim, R. Jain, and S. K. Lee, Machine learning and radiomic phenotyping of lower grade gliomas: Improving survival prediction, Eur. Radiol., vol. 30, no. 7, pp. 3834–3842, 2020.

Crossref Google Scholar
[27]

T. Ching, X. Zhu, and L. X. Garmire, Cox-nnet: An artificial neural network method for prognosis prediction of high-throughput omics data, PLoS Comput. Biol., vol. 14, no. 4, p. e1006076, 2018.

Crossref Google Scholar
[28]

Z. C. Li, J. Yan, S. Zhang, C. Liang, X. Lv, Y. Zou, H. Zhang, D. Liang, Z. Zhang, and Y. Chen, Glioma survival prediction from whole-brain MRI without tumor segmentation using deep attention network: A multicenter study, Eur. Radiol., vol. 32, no. 8, pp. 5719–5729, 2022.

Crossref Google Scholar
[29]

M. Meng, B. Gu, L. Bi, S. Song, D. D. Feng, and J. Kim, DeepMTS: Deep multi-task learning for survival prediction in patients with advanced nasopharyngeal carcinoma using pretreatment PET/CT, IEEE J. Biomed. Health Inform., vol. 26, no. 9, pp. 4497–4507, 2022.

Crossref Google Scholar
[30]
C. J. C. Burges, From ranknet to lambdarank to lambdamart: An overview, Tech. Rep. MSR-TR-2010-82, Microsoft Research, Redmond, WA, USA, 2010.
[31]
F. Isensee, P. F. Jaeger, S. A. A. Kohl, J. Petersen, and K. H. Maier-Hein, nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation, Nat. Methods, vol. 18, no. 2, pp. 203–211, 2021.
[32]
F. Milletari, N. Navab, and S. A. Ahmadi, V-net: Fully convolutional neural networks for volumetric medical image segmentation, in Proc. Fourth Int. Conf. 3D Vision (3DV), Stanford, CA, USA, 2016, pp. 565–571.
[33]

K. Chang, A. L. Beers, H. X. Bai, J. M. Brown, K. I. Ly, X. Li, J. T. Senders, V. K. Kavouridis, A. Boaro, C. Su, et al., Automatic assessment of glioma burden: A deep learning algorithm for fully automated volumetric and bidimensional measurement, Neuro-Oncology, vol. 21, no. 11, pp. 1412–1422, 2019.

Crossref Google Scholar
[34]

J. Peng, D. D. Kim, J. B. Patel, X. Zeng, J. Huang, K. Chang, X. Xun, C. Zhang, J. Sollee, J. Wu, et al., Deep learning-based automatic tumor burden assessment of pediatric high-grade gliomas, medulloblastomas, and other leptomeningeal seeding tumors, Neuro-Oncology, vol. 24, no. 2, pp. 289–299, 2022.

Crossref Google Scholar
[35]

D. R. Cox, Regression models and life-tables, J. R. Stat. Soc. Ser. B Stat. Methodol., vol. 34, no. 2, pp. 187–202, 1972.

Crossref Google Scholar
[36]

X. Li and Y. Meng, Survival analysis of immune-related lncRNA in low-grade glioma, BMC Cancer, vol. 19, no. 1, p. 813, 2019.

Crossref Google Scholar
[37]

J. Lao, Y. Chen, Z. C. Li, Q. Li, J. Zhang, J. Liu, and G. Zhai, A deep learning-based radiomics model for prediction of survival in glioblastoma multiforme, Sci. Rep., vol. 7, no. 1, p. 10353, 2017.

Crossref Google Scholar
[38]

K. Alleman, E. Knecht, J. Huang, L. Zhang, S. Lam, and M. DeCuypere, Multimodal deep learning-based prognostication in glioma patients: A systematic review, Cancers, vol. 15, no. 2, p. 545, 2023.

Crossref Google Scholar
[39]

X. Hu, E. Martinez-Ledesma, S. Zheng, H. Kim, F. Barthel, T. Jiang, K. R. Hess, and R. G. W. Verhaak, Multigene signature for predicting prognosis of patients with 1p19q co-deletion diffuse glioma, Neuro-Oncology, vol. 19, no. 6, pp. 786–795, 2017.

Crossref Google Scholar
[40]
L. Cheng, H. Wu, X. Zheng, N. Zhang, P. Zhao, R. Wang, Q. Wu, T. Liu, X. Yang, and Q. Geng, GPGPS: A robust prognostic gene pair signature of glioma ensembling IDH mutation and 1p/19q co-deletion, Bioinformatics, vol. 39, no. 1, p. btac850, 2023.
[41]

A. Chelliah, D. A. Wood, L. S. Canas, H. Shuaib, S. Currie, K. Fatania, R. Frood, C. Rowland-Hill, S. Thust, S. J. Wastling, et al., Glioblastoma and radiotherapy: A multicenter AI study for survival predictions from MRI (GRASP study), Neuro-Oncology, vol. 26, no. 6, pp. 1138–1151, 2024.

Crossref Google Scholar
[42]

J. Fu, K. Singhrao, X. Zhong, Y. Gao, S. X. Qi, Y. Yang, D. Ruan, and J. H. Lewis, An automatic deep learning–based workflow for glioblastoma survival prediction using preoperative multimodal MR images: A feasibility study, Adv. Radiat. Oncol., vol. 6, no. 5, p. 100746, 2021.

Crossref Google Scholar
[43]

M. Ramkumar, R. S. Kumar, R. Padmapriya, and S. Karthick, Brain tumor segmentation and survival time prediction using graph momentum fully convolutional network with modified Elman spike neural network, Int. J. Imag. Syst. Technol., vol. 34, no. 1, p. e23005, 2024.

Crossref Google Scholar
[44]

L. Zhou, Q. Ji, H. Peng, F. Chen, Y. Zheng, Z. Jiao, J. Gong, and W. Li, Automatic image segmentation and online survival prediction model of medulloblastoma based on machine learning, Eur. Radiol., vol. 34, no. 6, pp. 3644–3655, 2024.

Google Scholar
[45]

W. Wu, J. Yan, Y. Zhao, Q. Sun, H. Zhang, J. Cheng, D. Liang, Y. Chen, Z. Zhang, and Z. C. Li, Multi-task learning for concurrent survival prediction and semi-supervised segmentation of gliomas in brain MRI, Displays, vol. 78, p. 102402, 2023.

Crossref Google Scholar
[46]
V. Andrearczyk, P. Fontaine, V. Oreiller, J. Castelli, M. Jreige, J. O. Prior, and A. Depeursinge, Multi-task deep segmentation and radiomics for automatic prognosis in head and neck cancer, in Proc. Predictive Intelligence in Medicine : 4th International Workshop, PRIME 2021, Strasbourg, France, 2021, pp. 147–156.
[47]
R. Cipolla, Y. Gal, and A. Kendall, Multi-task learning using uncertainty to weigh losses for scene geometry and semantics, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 7482–7491.
[48]
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., An image is worth 16×16 words: Transformers for image recognition at scale, presented at International Conference on Learning Representations, Vienna, Austria, 2021.
[49]

Y. Zhou, H. Chen, Y. Li, Q. Liu, X. Xu, S. Wang, P. T. Yap, and D. Shen, Multi-task learning for segmentation and classification of tumors in 3D automated breast ultrasound images, Med. Image Anal., vol. 70, p. 101918, 2021.

Crossref Google Scholar
[50]

S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. S. Kirby, J. B. Freymann, K. Farahani, and C. Davatzikos, Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features, Sci. Data, vol. 4, p. 170117, 2017.

Crossref Google Scholar
[51]

K. Clark, B. Vendt, K. Smith, J. Freymann, J. Kirby, P. Koppel, S. Moore, S. Phillips, D. Maffitt, M. Pringle, et al., The cancer imaging archive (TCIA): Maintaining and operating a public information repository, J. Transl. Med., vol. 26, no. 6, pp. 1045–1057, 2013.

Google Scholar
[52]

M. Ceccarelli, F. P. Barthel, T. M. Malta, T. S. Sabedot, S. R. Salama, B. A. Murray, O. Morozova, Y. Newton, A. Radenbaugh, S. M. Pagnotta, et al., Molecular profiling reveals biologically discrete subsets and pathways of progression in diffuse glioma, Cell, vol. 164, no. 3, pp. 550–563, 2016.

Crossref Google Scholar
[53]

J. L. Katzman, U. Shaham, A. Cloninger, J. Bates, T. Jiang, and Y. Kluger, DeepSurv: Personalized treatment recommender system using a Cox proportional hazards deep neural network, BMC Med. Res. Methodol., vol. 18, no. 1, p. 24, 2018.

Crossref Google Scholar
[54]
X. Zhu, J. Yao, and J. Huang, Deep convolutional neural network for survival analysis with pathological images, in Proc. IEEE Int. Conf. Bioinformatics and Biomedicine (BIBM), Shenzhen, China, 2016, pp. 544–547.
[55]

P. Chunduru, J. J. Phillips, and A. M. Molinaro, Prognostic risk stratification of gliomas using deep learning in digital pathology images, Neuro Oncol. Adv., vol. 4, no. 1, p. vdac111, 2022.

Crossref Google Scholar
[56]
T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn, Gradient surgery for multi-task learning, arXiv preprint arXiv: 2001.06782v4, 2020.
[57]

The Cancer Genome Atlas Research Network, Comprehensive, integrative genomic analysis of diffuse lower-grade gliomas, N. Engl. J. Med., vol. 372, no. 26, pp. 2481–2498, 2015.

Crossref Google Scholar
[58]
A. Darlix, J. Deverdun, N. M. de Champfleur, F. Castan, S. Zouaoui, V. Rigau, M. Fabbro, Y. Yordanova, E. Le Bars, L. Bauchet, et al., IDH mutation and 1p19q codeletion distinguish two radiological patterns of diffuse low-grade gliomas, J. Neuro Oncol., vol. 133, no. 1, pp. 37–45, 2017.
[59]

H. Peng, J. Huo, B. Li, Y. Cui, H. Zhang, L. Zhang, and L. Ma, Predicting isocitrate dehydrogenase (IDH) mutation status in gliomas using multiparameter MRI radiomics features, J. Magn. Reson. Imag., vol. 53, no. 5, pp. 1399–1407, 2021.

Crossref Google Scholar
[60]

H. Zhou, K. Chang, H. X. Bai, B. Xiao, C. Su, W. L. Bi, P. J. Zhang, J. T. Senders, M. Vallières, V. K. Kavouridis, et al., Machine learning reveals multimodal MRI patterns predictive of isocitrate dehydrogenase and 1p/19q status in diffuse low- and high-grade gliomas, J. Neurooncol, vol. 142, no. 2, pp. 299–307, 2019.

Crossref Google Scholar
Big Data Mining and Analytics
Volume 8 Issue 2,
April 2025
Pages 364-382
DOI: 10.26599/BDMA.2024.9020083
Cite this article:
Cheng J, Kuang H, Yang S, et al. Segmentation-Guided Deep Learning for Glioma Survival Risk Prediction with Multimodal MRI. Big Data Mining and Analytics, 2025, 8(2): 364-382. https://doi.org/10.26599/BDMA.2024.9020083
Metrics & Citations  
Article History
Copyright
Rights and Permissions
Return