| Sign up

PDF (469.2 KB)

Cite

Collect

Submit Manuscript

Show Outline

Figures (3)

Fig. 1

Fig. 2

Fig. 3

Tables (3)

TableA 1

Table 1

Table 2

Open Access

Three-Branch BERT-Based Text Classification Network for Gastroscopy Diagnosis Text

Zhichao Wang^{¹^,²}, Xiangwei Zheng^{¹^,²}(), Jinsong Zhang^{¹^,²}, Mingzhe Zhang^{¹^,²}()

1School of Information Science and Engineering, Shandong Normal University, Jinan 250358, China

2State Key Laboratory of High-end Server & Storage Technology, Jinan 250013, China

Show Author Information

Abstract

During a hospital visit, a significant volume of Gastroscopy Diagnostic Text (GDT) data are produced, representing the unstructured gastric medical records of patients undergoing gastroscopy. As such, GDTs play a crucial role in evaluating the patient’s health, shaping treatment plans, and scheduling follow-up visits. However, given the free-text nature of GDTs, which lack a formal structure, physicians often find it challenging to extract meaningful insights from them. Furthermore, while deep learning has made significant strides in the medical domain, to our knowledge, there are not any readily available text-based pre-trained models tailored for GDT classification and analysis. To address this gap, we introduce a Bidirectional Encoder Representations from Transformers (BERT) based three-branch classification network tailored for GDTs. We leverage the robust representation capabilities of the BERT pre-trained model to deeply encode the texts. A unique three-branch decoder structure is employed to pinpoint lesion sites and determine cancer stages. Experimental outcomes validate the efficacy of our approach in GDT classification, with a precision of 0.993 and a recall of 0.784 in the early cancer category. In pinpointing cancer lesion sites, the weighted F1 score achieved was 0.849.

Keywords

gastroscopy diagnostic text Bidirectional Encoder Representations from Transformers (BERT)text classification

References

[1]

Y. Cui, G. Cheng, G. Tian, S. He, and Y. Yan, Secular trends in the mortality of gastrointestinal cancers across China, Japan, the US, and India: An age-period-cohort, joinpoint analyses, and Holt forecasts, Front. Public Health, vol. 10, p. 925011, 2022.

Crossref Google Scholar

[2]

C. Herrera-Pariente, S. Montori, J. Llach, A. Bofill, E. Albeniz, and L. Moreira, Biomarkers for gastric cancer screening and early diagnosis, Biomedicines, vol. 9, no. 10, p. 1448, 2021.

Crossref Google Scholar

[3]

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, Attention is all you need, in Proc. 31st Int. Conf. Neural Information Processing Systems, Long Beach, CA, USA, 2017, pp. 6000–6010.

[4]

J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv: 1810.04805, 2019.

[5]

W. Zaremba, I. Sutskever, and O. Vinyals, Recurrent neural network regularization, arXiv preprint arXiv: 1409.2329, 2014.

[6]

A. Graves, Long short-term memory, in Supervised Sequence Labelling with Recurrent Neural Networks, A. Graves, ed. Berlin, Germany: Springer, 2012, pp. 37–45.

Crossref

[7]

A. Onan and M. A. Toçoğlu, A term weighted neural language model and stacked bidirectional LSTM based framework for sarcasm identification, IEEE Access, vol. 9, pp. 7701–7722, 2021.

Crossref Google Scholar

[8]

W. Zhou, J. Du, and X. Ren, Improving BERT fine-tuning with embedding normalization, arXiv preprint arXiv: 1911.03918, 2019.

[9]

L. Jiang, C. Li, S. Wang, and L. Zhang, Deep feature weighting for naive Bayes and its application to text classification, Eng. Appl. Artif. Intell., vol. 52, pp. 26–39, 2016.

Crossref Google Scholar

[10]

T. Joachims, Making large scale SVM learning practical, https://api.semanticscholar.org/CorpusID:61116019, 1998.

[11]

J. R. Quinlan, Induction of decision trees, Mach. Learn., vol. 1, no. 1, pp. 81–106, 1986.

Crossref Google Scholar

[12]

L. Breiman, Random forests, Mach. Learn., vol. 45, no. 1, pp. 5–32, 2001.

Crossref Google Scholar

[13]

Y. Yang and X. Liu, A re-examination of text categorization methods, in Proc. 22nd Annual Int. ACM SIGIR Conf. Research and Development in Information Retrieval, Berkeley, CA, USA, 1999, pp. 42–49.

Crossref

[14]

S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, and J. Gao, Deep learning: Based text classification, ACM Comput. Surv., vol. 54, no. 3, pp. 1–40, 2022.

Crossref Google Scholar

[15]

M. Hughes, I. Li, S. Kotoulas, and T. Suzumura, Medical text classification using convolutional neural networks, Stud. Health Technol. Inform., vol. 235, pp. 246–250, 2017.

Google Scholar

[16]

A. Onan, Bidirectional convolutional recurrent neural network architecture with group-wise enhancement mechanism for text sentiment classification, J. King Saud Univ. Comput. Inf. Sci., vol. 34, no. 5, pp. 2098–2117, 2022.

Crossref Google Scholar

[17]

A. Onan, S. Korukoğlu, and H. Bulut, Ensemble of keyword extraction methods and classifiers in text classification, Expert Syst. Appl., vol. 57, pp. 232–247, 2016.

Crossref Google Scholar

[18]

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, RoBERTa: A robustly optimized BERT pretraining approach, arXiv preprint arXiv: 1907.11692, 2019.

[19]

A. Onan, Hierarchical graph-based text classification framework with contextual node embedding and BERT-based dynamic fusion, J. King Saud Univ. Comput. Inf. Sci., vol. 35, no. 7, p. 101610, 2023.

Crossref Google Scholar

[20]

A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, Improving language understanding by generative pre-training, https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf, 2018.

[21]

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, in Proc. 34th Int. Conf. Neural Information Processing Systems, Vancouver, Canada, 2020, pp. 1877–1901.

[22]

S. Ding, S. Hu, X. Li, Y. Zhang, and D. D. Wu, Leveraging multimodal semantic fusion for gastric cancer screening via hierarchical attention mechanism, IEEE Trans. Syst. Man Cybern. Syst., vol. 52, no. 7, pp. 4286–4299, 2022.

Crossref Google Scholar

[23]

Y. Liu, Fine-tune BERT for extractive summarization, arXiv preprint arXiv: 1903.10318, 2019.

[24]

N. Ketkar, Introduction to keras, in Deep Learning with Python, N. Ketkar, ed. Berkeley, CA, USA: Apress, 2017, pp. 97–111.

Crossref

International Journal of Crowd Science

Volume 8 Issue 1,
February 2024

Pages 56-63

DOI: 10.26599/IJCS.2023.9100031

Cite this article:

Wang Z, Zheng X, Zhang J, et al. Three-Branch BERT-Based Text Classification Network for Gastroscopy Diagnosis Text. International Journal of Crowd Science, 2024, 8(1): 56-63. https://doi.org/10.26599/IJCS.2023.9100031

Data category	Number of training sets	Number of validation sets	Total
Early-stage cancer	246	53	299
Progressive cancer	352	82	434
Other	1369	357	1726
Total	1967	492	2459

Model	Precision	Recall	F1	Early cancer recall	$F 1_{ω}$
CNN	0.934	0.926	0.930	0.496	—
LSTM	0.842	0.855	0.848	0.428	—
BERTCN (RoBERTa)	0.993	0.993	0.993	0.765	—
ML-BERTCN (RoBERTa)	—	—	—	—	0.724
GPT-3.5 turbo (API)	0.557	0.659	0.637	0.679	0.707
SIMO-BERTCN(RoBERTa)	0.962	0.933	0.947	0.679	0.591
Our (RoBERTa)	0.993	0.993	0.993	0.784	0.849

Algorithm 1　Keyword-based text slicing method for GDT
Input: gastroscope diagnosis text gdt
Output: text unit list text_unit_list
1 gdt = “Esophageal: The mucosa is smooth…”
2 stomach_sites = [“Esophageal”, “cardia”, “Gastric fundus”,　 “Gastric body”, “Gastric antrum”, “Duodenal bulb and descending”]
3 text_unit_list = []
// Split the original GDT text by periods and line breaks
4 gdt = split([“\n”, “.”], gdt)
5 for text_part in gdt:
6 　for site in stomach_sites:
7 　　if site in text_part:
// Text segment containing the intragastric site keyword
8 　　　text_unit_list.append(text_part) #
9 　　else if text_unit_list:
// The text segment does not contain the intragastric site 　keyword
10 　　　text_unit_list[−1].concat(text_part)
11 　　end if
12 　end for
13 end for