AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home International Journal of Crowd Science Article
PDF (469.2 KB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Open Access

Three-Branch BERT-Based Text Classification Network for Gastroscopy Diagnosis Text

Zhichao Wang1,2, Xiangwei Zheng1,2( ), Jinsong Zhang1,2, Mingzhe Zhang1,2( )
School of Information Science and Engineering, Shandong Normal University, Jinan 250358, China
State Key Laboratory of High-end Server & Storage Technology, Jinan 250013, China
Show Author Information

Abstract

During a hospital visit, a significant volume of Gastroscopy Diagnostic Text (GDT) data are produced, representing the unstructured gastric medical records of patients undergoing gastroscopy. As such, GDTs play a crucial role in evaluating the patient’s health, shaping treatment plans, and scheduling follow-up visits. However, given the free-text nature of GDTs, which lack a formal structure, physicians often find it challenging to extract meaningful insights from them. Furthermore, while deep learning has made significant strides in the medical domain, to our knowledge, there are not any readily available text-based pre-trained models tailored for GDT classification and analysis. To address this gap, we introduce a Bidirectional Encoder Representations from Transformers (BERT) based three-branch classification network tailored for GDTs. We leverage the robust representation capabilities of the BERT pre-trained model to deeply encode the texts. A unique three-branch decoder structure is employed to pinpoint lesion sites and determine cancer stages. Experimental outcomes validate the efficacy of our approach in GDT classification, with a precision of 0.993 and a recall of 0.784 in the early cancer category. In pinpointing cancer lesion sites, the weighted F1 score achieved was 0.849.

Keywords

gastroscopy diagnostic textBidirectional Encoder Representations from Transformers (BERT)text classification

References

[1]

Y. Cui, G. Cheng, G. Tian, S. He, and Y. Yan, Secular trends in the mortality of gastrointestinal cancers across China, Japan, the US, and India: An age-period-cohort, joinpoint analyses, and Holt forecasts, Front. Public Health, vol. 10, p. 925011, 2022.
Crossref Google Scholar
[2]

C. Herrera-Pariente, S. Montori, J. Llach, A. Bofill, E. Albeniz, and L. Moreira, Biomarkers for gastric cancer screening and early diagnosis, Biomedicines, vol. 9, no. 10, p. 1448, 2021.
Crossref Google Scholar
[3]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, Attention is all you need, in Proc. 31st Int. Conf. Neural Information Processing Systems, Long Beach, CA, USA, 2017, pp. 6000–6010.
[4]
J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv: 1810.04805, 2019.
[5]
W. Zaremba, I. Sutskever, and O. Vinyals, Recurrent neural network regularization, arXiv preprint arXiv: 1409.2329, 2014.
[6]
A. Graves, Long short-term memory, in Supervised Sequence Labelling with Recurrent Neural Networks, A. Graves, ed. Berlin, Germany: Springer, 2012, pp. 37–45.
Crossref
[7]

A. Onan and M. A. Toçoğlu, A term weighted neural language model and stacked bidirectional LSTM based framework for sarcasm identification, IEEE Access, vol. 9, pp. 7701–7722, 2021.
Crossref Google Scholar
[8]
W. Zhou, J. Du, and X. Ren, Improving BERT fine-tuning with embedding normalization, arXiv preprint arXiv: 1911.03918, 2019.
[9]

L. Jiang, C. Li, S. Wang, and L. Zhang, Deep feature weighting for naive Bayes and its application to text classification, Eng. Appl. Artif. Intell., vol. 52, pp. 26–39, 2016.
Crossref Google Scholar
[10]
T. Joachims, Making large scale SVM learning practical, https://api.semanticscholar.org/CorpusID:61116019, 1998.
[11]

J. R. Quinlan, Induction of decision trees, Mach. Learn., vol. 1, no. 1, pp. 81–106, 1986.
Crossref Google Scholar
[12]

L. Breiman, Random forests, Mach. Learn., vol. 45, no. 1, pp. 5–32, 2001.
Crossref Google Scholar
[13]
Y. Yang and X. Liu, A re-examination of text categorization methods, in Proc. 22nd Annual Int. ACM SIGIR Conf. Research and Development in Information Retrieval, Berkeley, CA, USA, 1999, pp. 42–49.
Crossref
[14]

S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, and J. Gao, Deep learning: Based text classification, ACM Comput. Surv., vol. 54, no. 3, pp. 1–40, 2022.
Crossref Google Scholar
[15]

M. Hughes, I. Li, S. Kotoulas, and T. Suzumura, Medical text classification using convolutional neural networks, Stud. Health Technol. Inform., vol. 235, pp. 246–250, 2017.
Google Scholar
[16]

A. Onan, Bidirectional convolutional recurrent neural network architecture with group-wise enhancement mechanism for text sentiment classification, J. King Saud Univ. Comput. Inf. Sci., vol. 34, no. 5, pp. 2098–2117, 2022.
Crossref Google Scholar
[17]

A. Onan, S. Korukoğlu, and H. Bulut, Ensemble of keyword extraction methods and classifiers in text classification, Expert Syst. Appl., vol. 57, pp. 232–247, 2016.
Crossref Google Scholar
[18]
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, RoBERTa: A robustly optimized BERT pretraining approach, arXiv preprint arXiv: 1907.11692, 2019.
[19]

A. Onan, Hierarchical graph-based text classification framework with contextual node embedding and BERT-based dynamic fusion, J. King Saud Univ. Comput. Inf. Sci., vol. 35, no. 7, p. 101610, 2023.
Crossref Google Scholar
[20]
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, Improving language understanding by generative pre-training, https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf, 2018.
[21]
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, in Proc. 34th Int. Conf. Neural Information Processing Systems, Vancouver, Canada, 2020, pp. 1877–1901.
[22]

S. Ding, S. Hu, X. Li, Y. Zhang, and D. D. Wu, Leveraging multimodal semantic fusion for gastric cancer screening via hierarchical attention mechanism, IEEE Trans. Syst. Man Cybern. Syst., vol. 52, no. 7, pp. 4286–4299, 2022.
Crossref Google Scholar
[23]
Y. Liu, Fine-tune BERT for extractive summarization, arXiv preprint arXiv: 1903.10318, 2019.
[24]
N. Ketkar, Introduction to keras, in Deep Learning with Python, N. Ketkar, ed. Berkeley, CA, USA: Apress, 2017, pp. 97–111.
Crossref
International Journal of Crowd Science
Volume 8 Issue 1,
March 2024
Pages 56-63
DOI: 10.26599/IJCS.2023.9100031
Cite this article:
Wang Z, Zheng X, Zhang J, et al. Three-Branch BERT-Based Text Classification Network for Gastroscopy Diagnosis Text. International Journal of Crowd Science, 2024, 8(1): 56-63. https://doi.org/10.26599/IJCS.2023.9100031

353

Views

28

Downloads

1

Crossref

1

Scopus

Altmetrics
Received: 08 September 2023
Revised: 02 November 2023
Accepted: 16 November 2023
Published: 27 February 2024
© The author(s) 2024.

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).
Return