Home Big Data Mining and Analytics Article
PDF (23.1 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript
Show Outline
Figures (11)

Show 2 more figures Hide 2 figures
Tables (9)
Table 1
Table 2
Table 3
Table 4
Table 5
Show 4 more tables Hide 4 tables
Open Access

SeaConvNeXt: A Lightweight Two-Branch Network Architecture for Efficient Prediction of Specific IHC Proteins and Antigens on Hematoxylin and Eosin (H&E) Images

School of Computer Science, Shaanxi Normal University, Xi’an 710119, China
Department of Pathology, Shaanxi Provincial Hospital of Traditional Chinese Medicine, Xi’an 710003, China
Wallace H. Coulter Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta 30322, GA, USA, and also with Joseph Maxwell Cleland Atlanta VA Medical Center, Decatur 30033, GA, USA
Department of Radiology and Medical Research Institute, Guangdong Provincial People’s Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou 510080, China, and also with Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application, Guangzhou 510080, China
Show Author Information

Abstract

Immunohistochemistry (IHC) is a vital technique for detecting specific proteins and antigens in tissue sections using antibodies, aiding in the analysis of tumor growth and metastasis. However, IHC is costly and time-consuming, making it challenging to implement on a large scale. To address this issue, we introduce a method that enables virtual IHC staining directly on Hematoxylin and Eosin (H&E) images. Firstly, we have developed a novel registration technique, called Bi-stage Registration based on density Clustering (BiReC), to enhance the registration efficiency between H&E and IHC images. This method involves automatically generating numerous Regions Of Interest (ROI) labels on the H&E image for model training, with the labels being determined by the intensity of IHC staining. Secondly, we propose a novel two-branch network architecture, called SeaConvNeXt, which integrates a lightweight Squeeze-Enhanced Axial (SEA) attention mechanism to efficiently extract and fuse multi-level local and global features from H&E images for direct prediction of specific proteins and antigens. The SeaConvNeXt consists of a ConvNeXt branch and a global fusion branch. The ConvNeXt branch extracts multi-level local features at four stages, while the global fusion branch, including an SEA Transformer module and three global blocks, is designed for global feature extraction and multiple feature fusion. Our experiments demonstrate that SeaConvNeXt outperforms current state-of-the-art methods on two public datasets with corresponding IHC and H&E images, achieving an AUC of 90.7% on the HER2SC dataset and 82.5% on the CRC dataset. These results suggest that SeaConvNeXt has great potential for predicting virtual IHC staining on H&E images.

Keywords

Immunohistochemistry (IHC)Bi-stage Registration based on density Clustering (BiReC)automatic label generationSeaConvNeXtattention mechanismmulti-level local and global featuresvirtual IHC staning prediction

References

[1]

J. Griffin and D. Treanor, Digital pathology in clinical use: Where are we now and what is holding us back, Histopathology, vol. 70, no. 1, pp. 134–145, 2017.

Crossref Google Scholar
[2]

S. S. Raab, D. M. Grzybicki, J. E. Janosky, R. J. Zarbo, F. A. Meier, C. Jensen, and S. J. Geyer, Clinical impact and frequency of anatomic pathology errors in cancer diagnoses, Cancer, vol. 104, no. 10, pp. 2205–2213, 2005.

Crossref Google Scholar
[3]

J. Rodriguez-Canales, F. C. Eberle, E. S. Jaffe, and M. R. Emmert-Buck, Why is it crucial to reintegrate pathology into cancer research, BioEssays, vol. 33, no. 7, pp. 490–498, 2011.

Crossref Google Scholar
[4]

M. Lacroix-Triki, S. Mathoulin-Pelissier, J.-P. Ghnassia, G. Macgrogan, A. Vincent-Salomon, V. Brouste, M.-C. Mathieu, P. Roger, F. Bibeau, J. Jacquemier, et al., High inter-observer agreement in immunohistochemical evaluation of HER-2/neu expression in breast cancer: A multicentre GEFPICS study, Eur. J. Cancer, vol. 42, no. 17, pp. 2946–2953, 2006.

Crossref Google Scholar
[5]

G. Yu, K. Sun, C. Xu, X.-H. Shi, C. Wu, T. Xie, R.-Q. Meng, X.-H. Meng, K.-S. Wang, H.-M. Xiao, et al., Accurate recognition of colorectal cancer with semi-supervised deep learning on pathological images, Nat. Commun., vol. 12, p. 6311, 2021.

Crossref Google Scholar
[6]

S. Sayed, R. Lukande, and K. A. Fleming, Providing pathology support in low-income countries, J. Glob. Oncol., vol. 1, no. 1, pp. 3–6, 2015.

Crossref Google Scholar
[7]

F. Yuan, Z. Zhang, and Z. Fang, An effective CNN and Transformer complementary network for medical image segmentation, Pattern Recognit., vol. 136, p. 109228, 2023.

Crossref Google Scholar
[8]

Y. Chen, Y. Zhou, G. Chen, Y. Guo, Y. Lv, M. Ma, Z. Pei, and Z. Sun, Segmentation of breast tubules in H&E images based on a DKS-DoubleU-net model, BioMed Res. Int., vol. 2022, p. 2961610, 2022.

Crossref Google Scholar
[9]
X. Li, A. Jiang, S. Wang, F. Li, and S. Yan, CTBP-Net: Lung nodule segmentation model based on the cross-transformer and bidirectional pyramid, Biomed. Signal Process. Contr., vol. 82, p. 104528, 2023.
Crossref
[10]

H. Xu, L. Liu, X. Lei, M. Mandal, and C. Lu, An unsupervised method for histological image segmentation based on tissue cluster level graph cut, Comput. Med. Imag. Graph., vol. 93, p. 101974, 2021.

Crossref Google Scholar
[11]

T. Mathew, B. Ajith, J. R. Kini, and J. Rajan, Deep learning-based automated mitosis detection in histopathology images for breast cancer grading, Int. J. Imag. Syst. Technol., vol. 32, no. 4, pp. 1192–1208, 2022.

Crossref Google Scholar
[12]

G. Xi, Q. Wang, H. Zhan, D. Kang, Y. Liu, T. Luo, M. Xu, Q. Kong, L. Zheng, G. Chen, et al., Automated classification of breast cancer histologic grade using multiphoton microscopy and generative adversarial networks, J. Phys. D: Appl. Phys., vol. 56, no. 1, p. 015401, 2023.

Crossref Google Scholar
[13]

Y. Wang, X. Lei, and Y. Pan, Predicting microbe-disease association based on heterogeneous network and global graph feature learning, Chin. J Electronics, vol. 31, no. 2, pp. 345–353, 2022.

Crossref Google Scholar
[14]

C. Fan, X. Lei, L. Guo, and A. Zhang, Predicting the associations between microbes and diseases by integrating multiple data sources and path-based HeteSim scores, Neurocomputing, vol. 323, pp. 76–85, 2019.

Crossref Google Scholar
[15]
H. Meng, X. Liu, J. Niu, Y. Wang, J. Liao, Q. Li, and C. Chen, DGANet: A dual global attention neural network for breast lesion detection in ultrasound images, Ultrasound Med. Biol., vol. 49, no. 1, pp. 31–44, 2023.
Crossref
[16]

S. Khaliliboroujeni, X. He, W. Jia, and S. Amirgholipour, End-to-end metastasis detection of breast cancer from histopathology whole slide images, Comput. Med. Imag. Graph., vol. 102, p. 102136, 2022.

Crossref Google Scholar
[17]

H. Mkindu, L. Wu, and Y. Zhao, Lung nodule detection in chest CT images based on vision transformer network with Bayesian optimization, Biomed. Signal Process. Contr., vol. 85, p. 104866, 2023.

Crossref Google Scholar
[18]

N. Zhang, Y.-X. Cai, Y.-Y. Wang, Y.-T. Tian, X.-L. Wang, and B. Badami, Skin cancer diagnosis based on optimized convolutional neural network, Artif. Intell. Med., vol. 102, p. 101756, 2020.

Crossref Google Scholar
[19]

W. Long, Y. Yang, and H.-B. Shen, ImPLoc: A multi-instance deep learning model for the prediction of protein subcellular localization based on immunohistochemistry images, Bioinformatics, vol. 36, no. 7, pp. 2244–2250, 2020.

Crossref Google Scholar
[20]

J.-X. Hu, Y. Yang, Y.-Y. Xu, and H.-B. Shen, Incorporating label correlations into deep neural networks to classify protein subcellular location patterns in immunohistochemistry images, Proteins Struct. Funct. Bioinform., vol. 90, no. 2, pp. 493–503, 2022.

Crossref Google Scholar
[21]

G. Shamai, Y. Binenbaum, R. Slossberg, I. Duek, Z. Gil, and R. Kimmel, Artificial intelligence algorithms to assess hormonal status from tissue microarrays in patients with breast cancer, JAMA Netw. Open, vol. 2, no. 7, p. e197700, 2019.

Crossref Google Scholar
[22]

A. Su, H. Lee, X. Tan, C. J. Suarez, N. Andor, Q. Nguyen, and H. P. Ji, A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images, NPJ Precis. Onc., vol. 6, no. 1, p. 14, 2022.

Crossref Google Scholar
[23]
L. Theelke, F. Wilm, C. Marzahl, C. A. Bertram, R. Klopfleisch, A. Maier, M. Aubreville, and K. Breininger, Iterative cross-scanner registration for whole slide images, in Proc. IEEE/CVF Int. Conf. Computer Vision Workshops (ICCVW), Montreal, Canada, 2021, pp. 582–590.
Crossref
[24]
A. Shafique, M. Babaie, M. Sajadi, A. Batten, S. Skdar, and H. R. Tizhoosh, Automatic multi-stain registration of whole slide images in histopathology, in Proc. 43rd Annual Int. Conf. IEEE Engineering in Medicine & Biology Society (EMBC), Virtual Event, 2021, pp. 3622–3625.
Crossref
[25]

L. Venet, S. Pati, M. D. Feldman, M. P. Nasrallah, P. Yushkevich, and S. Bakas, Accurate and robust alignment of differently stained histologic images based on greedy diffeomorphic registration, Appl. Sci., vol. 11, no. 4, p. 1892, 2021.

Crossref Google Scholar
[26]

J. Yao, X. Zhu, J. Jonnagaddala, N. Hawkins, and J. Huang, Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks, Med. Image Anal., vol. 65, p. 101789, 2020.

Crossref Google Scholar
[27]

M. Z. Hoque, A. Keskinarkaus, P. Nyberg, T. Mattila, and T. Seppänen, Whole slide image registration via multi-stained feature matching, Comput. Biol. Med., vol. 144, p. 105301, 2022.

Crossref Google Scholar
[28]
P. Liu, F. Wang, G. Teodoro, and J. Kong, Histopathology image registration by integrated texture and spatial proximity based landmark selection and modification, in Proc. IEEE 18th Int. Symp. on Biomedical Imaging (ISBI), Nice, France, 2021, pp. 1827–1830.
Crossref
[29]

L. Ge, X. Wei, Y. Hao, J. Luo, and Y. Xu, Unsupervised histological image registration using structural feature guided convolutional neural network, IEEE Trans. Med. Imag., vol. 41, no. 9, pp. 2414–2431, 2022.

Crossref Google Scholar
[30]
J. Lotz, N. Weiss, J. van der Laak, and S. Heldmann, Comparison of consecutive and re-stained sections for image registration in histopathology, arXiv preprint arXiv: 2106.13150, 2021.
[31]
S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, CBAM: Convolutional block attention module, in Proc. European Conference on Computer Vision (ECCV 2018), doi: 10.1007/978-3-030-01234-2_1.
Crossref
[32]

Y. Zhang, L. Luo, Q. Dou, and P.-A. Heng, Triplet attention and dual-pool contrastive learning for clinic-driven multi-label medical image classification, Med. Image Anal., vol. 86, p. 102772, 2023.

Crossref Google Scholar
[33]

J. Cheng, S. Tian, L. Yu, C. Gao, X. Kang, X. Ma, W. Wu, S. Liu, and H. Lu, ResGANet: Residual Group attention network for medical image classification and segmentation, Med. Image Anal., vol. 76, p. 102313, 2022.

Crossref Google Scholar
[34]

M. Karri, C. S. R. Annavarapu, and U. R. Acharya, Explainable multi-module semantic guided attention based network for medical image segmentation, Comput. Biol. Med., vol. 151, p. 106231, 2022.

Crossref Google Scholar
[35]
Q. Wan, Z. Huang, J. Lu, G. Yu, and L. Zhang, SeaFormer++: Squeeze-enhanced axial transformer for mobile visual recognition, arXiv preprint arXiv: 2301.13156, 2023.
[36]

R. Yan, Q. He, Y. Liu, P. Ye, L. Zhu, S. Shi, J. Gou, Y. He, T. Guan, and G. Zhou, Unpaired virtual histological staining using prior-guided generative adversarial networks, Comput. Med. Imag. Graph., vol. 105, p. 102185, 2023.

Crossref Google Scholar
[37]
S. Liu, C. Zhu, F. Xu, X. Jia, Z. Shi, and M. Jin, BCI: Breast cancer immunohistochemical image generation through pyramid Pix2pix, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), New Orleans, LA, USA, 2022, pp. 1814–1823.
Crossref
[38]
R. Zhang, Y. Cao, Y. Li, Z. Liu, J. Wang, J. He, C. Zhang, X. Sui, P. Zhang, L. Cui, et al., MVFStain: Multiple virtual functional stain histopathology images generation based on specific domain mapping, Med. Image Anal., vol. 80, p. 102520, 2022.
Crossref
[39]

P. Gamble, R. Jaroensri, H. Wang, F. Tan, M. Moran, T. Brown, I. Flament-Auvigne, E. A. Rakha, M. Toss, D. J. Dabbs, et al., Determining breast cancer biomarker status and associated morphological features using deep learning, Commun. Med., vol. 1, no. 1, p. 14, 2021.

Crossref Google Scholar
[40]
R. R. Rawat, I. Ortega, P. Roy, F. Sha, D. Shibata, D. Ruderman, and D. B. Agus, Deep learned tissue “fingerprints” classify breast cancers by ER/PR/Her2 status from H&E images, Scientific Reports, vol. 10, p. 7275, 2020.
Crossref
[41]
J. Chen, Y. Lu, Q. Yu, X. Luo, E. Adeli, Y. Wang, L. Lu, A. L. Yuille, and Y. Zhou, TransUNet: Transformers make strong encoders for medical image segmentation, arXiv preprint arXiv:2102.04306, 2021.
[42]
Y. Zhang, H. Liu, and Q. Hu, Transfuse: Fusing transformers and CNNs for medical image segmentation, in Proc. 24th Int. Conf. Medical Image Computing and Computer Assisted Intervention – MICCAI, Strasbourg, France, 2021, pp. 14 –24.
Crossref
[43]
Y. Chen, X. Dai, D. Chen, M. Liu, X. Dong, L. Yuan, and Z. Liu, Mobile-former: Bridging MobileNet and transformer, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 5270–5279.
Crossref
[44]
Z. Peng, W. Huang, S. Gu, L. Xie, Y. Wang, J. Jiao, and Q. Ye, Conformer: Local features coupling global representations for visual recognition, in Proc. IEEE/CVF Int. Conf. Computer Vision (ICCV), Montreal, Canada, 2021, pp. 357–366.
Crossref
[45]
Z. Zhang, G. Sun, K. Zheng, J.-K. Yang, X.-R. Zhu, and Y. Li, TC-Net: A joint learning framework based on CNN and vision transformer for multi-lesion medical images segmentation, Comput. Biol. Med., vol. 161, p. 106967, 2023.
Crossref
[46]

A. Vahadane, T. Peng, A. Sethi, S. Albarqouni, L. Wang, M. Baust, K. Steiger, A. M. Schlitter, I. Esposito, and N. Navab, Structure-preserving color normalization and sparse stain separation for histological images, IEEE Trans. Med. Imag., vol. 35, no. 8, p. 1962–1971, 2016.

Crossref Google Scholar
[47]

A. C. Ruifrok and D. A. Johnston, Quantification of histochemical staining by color deconvolution, Anal. Quant. Cytol. Histol., vol. 23, no. 4, pp. 291–299, 2001.

Google Scholar
[48]

F. Shamshad, S. Khan, S. W. Zamir, M. H. Khan, M. Hayat, F. S. Khan, and H. Fu, Transformers in medical imaging: A survey, Med. Image Anal., vol. 88, p. 102802, 2023.

Crossref Google Scholar
[49]
Q. He, Q. Yang, and M. Xie, HCTNet: A hybrid CNN-transformer network for breast ultrasound image segmentation, Comput. Biol. Med., vol. 155, p. 106629, 2023.
Crossref
[50]
J. M. J. Valanarasu, P. Oza, I. Hacihaliloglu, and V. M. Patel, Medical transformer: Gated axial-attention for medical image segmentation, in Proc. 24th Int. Conf. Medical Image Computing and Computer Assisted Intervention – MICCAI, Strasbourg, France, 2021, pp. 36–46.
Crossref
[51]
Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, A ConvNet for the 2020s, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 11966–11976.
Crossref
[52]
S. Ioffe, Batch renormalization: Towards reducing minibatch dependence in batch-normalized models, arXiv preprint arXiv: 1702.03275, 2017.
[53]
J. L. Ba, J. R. Kiros, and G. E. Hinton, Layer normalization, arXiv preprint arXiv: 1607.06450, 2016.
[54]
T. Qaiser, A. Mukherjee, C. R. Pb, S. D. Munugoti, V. Tallam, T. Pitkäaho, T. Lehtimäki, T. Naughton, M. Berseth, A. Pedraza, et al., HER2 challenge contest: A detailed assessment of automated HER2 scoring algorithms in whole slide images of breast cancer tissues, Histopathology, vol. 72, no. 2, pp. 227–238, 2018.
Crossref
[55]

Y. Li, T. Yao, Y. Pan, and T. Mei, Contextual transformer networks for visual recognition, IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 2, pp. 1489–1500, 2023.

Crossref Google Scholar
[56]
Y. Liu, Z. Shao, and N. Hoffmann, Global attention mechanism: Retain information to enhance channel-spatial interactions, arXiv preprint arXiv: 2112.05561, 2021.
[57]
K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 770–778.
Crossref
[58]
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, MobileNetV2: Inverted residuals and linear bottlenecks, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 4510–4520.
Crossref
[59]
K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556, 2014.
[60]

B. Fu, M. Zhang, J. He, Y. Cao, Y. Guo, and R. Wang, StoHisNet: A hybrid multi-classification model with CNN and Transformer for gastric pathology images, Comput. Meth. Programs Biomed., vol. 221, p. 106924, 2022.

Crossref Google Scholar
[61]
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, Swin transformer: Hierarchical vision transformer using shifted windows, in Proc. IEEE/CVF Int. Conf. Computer Vision (ICCV), Montreal, Canada, 2021, pp. 9992–10002.
Crossref
[62]

Y. Liu, X. Li, A. Zheng, X. Zhu, S. Liu, M. Hu, Q. Luo, H. Liao, M. Liu, Y. He, et al., Predict Ki-67 positive cells in H&E-stained images using deep learning independently from IHC-stained images, Front. Mol. Biosci., vol. 7, p. 183, 2020.

Crossref Google Scholar
[63]

S. H. Lee, I. H. Song, and H.-J. Jang, Feasibility of deep learning-based fully automated classification of microsatellite instability in tissue slides of colorectal cancer, Int. J. Cancer, vol. 149, no. 3, pp. 728–740, 2021.

Crossref Google Scholar
[64]

R. J. Chen, T. Ding, M. Y. Lu, D. F. K. Williamson, G. Jaume, A. H. Song, B. Chen, A. Zhang, D. Shao, M. Shaban, et al., Towards a general-purpose foundation model for computational pathology, Nat. Med., vol. 30, no. 3, pp. 850–862, 2024.

Crossref Google Scholar
[65]
D. C. Bui, B. Song, K. Kim, and J. T. Kwak, DAX-Net: A dual-branch dual-task adaptive cross-weight feature fusion network for robust multi-class cancer classification in pathology images, Comput. Meth. Programs Biomed., vol. 248, p. 108112, 2024.
Crossref
[66]

X. Huo, G. Sun, S. Tian, Y. Wang, L. Yu, J. Long, W. Zhang, and A. Li, HiFuse: Hierarchical multi-scale feature fusion network for medical image classification, Biomed. Signal Process. Contr., vol. 87, p. 105534, 2024.

Crossref Google Scholar
[67]

L. Van Der Maaten and G. Hinton, Visualizing data using t-SNE, J. Mach. Learn. Res., vol. 9, pp. 2579–2625, 2008.

Google Scholar
[68]
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, Grad-CAM: Visual explanations from deep networks via gradient-based localization, in Proc. IEEE Int. Conf. Computer Vision (ICCV), Venice, Italy, 2017, pp. 618–626.
Crossref
Big Data Mining and Analytics
Volume 7 Issue 4,
December 2024
Pages 1212-1236
DOI: 10.26599/BDMA.2024.9020057
Cite this article:
Chen Y, Chen G, Shi G, et al. SeaConvNeXt: A Lightweight Two-Branch Network Architecture for Efficient Prediction of Specific IHC Proteins and Antigens on Hematoxylin and Eosin (H&E) Images. Big Data Mining and Analytics, 2024, 7(4): 1212-1236. https://doi.org/10.26599/BDMA.2024.9020057
Metrics & Citations  
Article History
Copyright
Rights and Permissions
Return

Distribution of datasets.

DatasetSubset (Number of WSIs)Positive (Number of tiles)Negative (Number of tiles)
CRCTrain (n = 20)1685821724
Valid (n = 5)45958180
Test (n = 5)649718783
HER2SCTrain (n = 17)1693129227
Valid (n = 5)37715840
Test (n = 5)36406067

Comparison of time required for different registration methods on two experimental datasets.

DatasetPair (number of WSIs)HEMnet registration (min)Fast BiReC registration (min)
CRC25227.82100.75
HER2SC27315.9392.93

Performance comparison of SeaConvNeXt with different image patch sizes.

DatasetPatch size (pixel × pixel)AUCACCF1Training time/epoch (min)
CRC336 × 3360.8070.6520.5293
224 × 2240.8250.7400.6528
112 × 1120.8380.6680.51427
HER2SC336 × 3360.8980.7530.6515
224 × 2240.9070.8220.8225
112 × 1120.8610.7660.74240

Comparison of parameters and FLOPs for SeaConvNeXt network with different attention mechanisms (best results are in bold).

ModelInput size (pixel × pixel)Number of params (×107)FLOPs (×109)
SeaConvNeXt (Transformer)224 × 2242.9647.84
SeaConvNeXt (COT)224 × 2243.52715.4
SeaConvNeXt (GAM)224 × 2244.61615.8
SeaConvNeXt (SEA)224 × 2242.8107.46

Performance comparison of the SeaConvNeXt networks with different attention mechanisms under different batch sizes on the two datasets (best results are in bold).

BatchsizeModelHER2SC datasetCRC dataset
AUCACCF1Time (min)AUCACCF1Time (min)
8SeaConvNeXt (SEA)0.8990.8160.798100.8050.6810.5615
SeaConvNeXt (GAM)0.9020.8110.794100.8150.690.56916
SeaConvNeXt (COT)0.9010.8170.803140.8280.7880.62518
SeaConvNeXt (Transformer)0.8950.7990.77312−−−−
16SeaConvNeXt (SEA)0.9070.8270.81870.7910.670.5459
SeaConvNeXt (GAM)−−−−−−−−
SeaConvNeXt (COT)−−−−−−−−
SeaConvNeXt (Transformer)−−−−−−−−
32SeaConvNeXt (SEA)0.9070.8220.82250.8250.740.6528
SeaConvNeXt (GAM)−−−−−−−−
SeaConvNeXt (COT)−−−−−−−−
SeaConvNeXt (Transformer)−−−−−−−−

Classification performance on the HER2SC dataset (best results are in bold).

ModelAUCACCF1
ResNet340.8870.8020.776
ResNet500.8930.8090.789
ResNet1010.8940.810.808
MobilenetV20.8890.8110.79
VGG160.8890.8090.804
Swin-Transformer0.8910.7950.769
Liu et al.[62]0.8890.7750.731
Lee et al.[63]0.8960.790.756
StoHisNet0.8970.8140.808
ConvNeXt0.8920.8030.776
UNI0.90.8190.818
DAX-Net0.8970.8110.79
HiFuse0.8960.7910.757
SeaConvNeXt0.9070.8220.822

Classification performance on the CRC dataset (best results are in bold).

ModelAUCACCF1
ResNet340.7720.7270.561
ResNet500.7980.7370.613
ResNet1010.8070.7260.614
MobilenetV20.760.6410.612
VGG160.8120.5330.491
Swin-Transformer0.7880.7380.609
Liu et al.[62]0.7950.6730.542
Lee et al.[63]0.8140.6970.578
StoHisNet0.7990.6710.548
ConvNeXt0.8130.7380.633
UNI0.7680.6180.471
DAX-Net0.8150.6540.558
HiFuse0.8180.6830.572
SeaConvNeXt0.8250.740.652

Performances of SeaConvNeXt with different components on the HER2SC dataset.

AblationModelAUCACCF1
Fusion stageSea_s10.9010.8130.794
Sea_s20.9020.8190.813
Sea_s30.8990.8110.803
Built-in moduleSea_nos0.8890.8010.789
Sea_noc0.8990.8050.791
CNN backboneResNet0.9000.8180.803
Parc-ConvNeXt0.8980.8170.805
BranchLocal0.8910.7840.749
Global0.9040.8200.813
OursSeaConvNeXt0.9070.8220.822

Performances of SeaConvNeXt with different components on the CRC dataset.

AblationModelAUCACCF1
Fusion stageSea_s10.8080.6990.570
Sea_s20.8120.6780.593
Sea_s30.8180.7230.593
Built-in moduleSea_nos0.7990.7110.567
Sea_noc0.8220.6990.578
CNN backboneResNet0.8160.7240.590
Parc-ConvNeXt0.8170.7130.581
BranchLocal0.8120.7210.591
Global0.8190.7330.598
OursSeaConvNeXt0.8250.7400.652