Home Big Data Mining and Analytics Article
PDF (23.1 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript
Open Access

SeaConvNeXt: A Lightweight Two-Branch Network Architecture for Efficient Prediction of Specific IHC Proteins and Antigens on Hematoxylin and Eosin (H&E) Images

School of Computer Science, Shaanxi Normal University, Xi’an 710119, China
Department of Pathology, Shaanxi Provincial Hospital of Traditional Chinese Medicine, Xi’an 710003, China
Wallace H. Coulter Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta 30322, GA, USA, and also with Joseph Maxwell Cleland Atlanta VA Medical Center, Decatur 30033, GA, USA
Department of Radiology and Medical Research Institute, Guangdong Provincial People’s Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou 510080, China, and also with Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application, Guangzhou 510080, China
Show Author Information

Abstract

Immunohistochemistry (IHC) is a vital technique for detecting specific proteins and antigens in tissue sections using antibodies, aiding in the analysis of tumor growth and metastasis. However, IHC is costly and time-consuming, making it challenging to implement on a large scale. To address this issue, we introduce a method that enables virtual IHC staining directly on Hematoxylin and Eosin (H&E) images. Firstly, we have developed a novel registration technique, called Bi-stage Registration based on density Clustering (BiReC), to enhance the registration efficiency between H&E and IHC images. This method involves automatically generating numerous Regions Of Interest (ROI) labels on the H&E image for model training, with the labels being determined by the intensity of IHC staining. Secondly, we propose a novel two-branch network architecture, called SeaConvNeXt, which integrates a lightweight Squeeze-Enhanced Axial (SEA) attention mechanism to efficiently extract and fuse multi-level local and global features from H&E images for direct prediction of specific proteins and antigens. The SeaConvNeXt consists of a ConvNeXt branch and a global fusion branch. The ConvNeXt branch extracts multi-level local features at four stages, while the global fusion branch, including an SEA Transformer module and three global blocks, is designed for global feature extraction and multiple feature fusion. Our experiments demonstrate that SeaConvNeXt outperforms current state-of-the-art methods on two public datasets with corresponding IHC and H&E images, achieving an AUC of 90.7% on the HER2SC dataset and 82.5% on the CRC dataset. These results suggest that SeaConvNeXt has great potential for predicting virtual IHC staining on H&E images.

Keywords

Immunohistochemistry (IHC)Bi-stage Registration based on density Clustering (BiReC)automatic label generationSeaConvNeXtattention mechanismmulti-level local and global featuresvirtual IHC staning prediction

References

[1]

J. Griffin and D. Treanor, Digital pathology in clinical use: Where are we now and what is holding us back, Histopathology, vol. 70, no. 1, pp. 134–145, 2017.

Crossref Google Scholar
[2]

S. S. Raab, D. M. Grzybicki, J. E. Janosky, R. J. Zarbo, F. A. Meier, C. Jensen, and S. J. Geyer, Clinical impact and frequency of anatomic pathology errors in cancer diagnoses, Cancer, vol. 104, no. 10, pp. 2205–2213, 2005.

Crossref Google Scholar
[3]

J. Rodriguez-Canales, F. C. Eberle, E. S. Jaffe, and M. R. Emmert-Buck, Why is it crucial to reintegrate pathology into cancer research, BioEssays, vol. 33, no. 7, pp. 490–498, 2011.

Crossref Google Scholar
[4]

M. Lacroix-Triki, S. Mathoulin-Pelissier, J.-P. Ghnassia, G. Macgrogan, A. Vincent-Salomon, V. Brouste, M.-C. Mathieu, P. Roger, F. Bibeau, J. Jacquemier, et al., High inter-observer agreement in immunohistochemical evaluation of HER-2/neu expression in breast cancer: A multicentre GEFPICS study, Eur. J. Cancer, vol. 42, no. 17, pp. 2946–2953, 2006.

Crossref Google Scholar
[5]

G. Yu, K. Sun, C. Xu, X.-H. Shi, C. Wu, T. Xie, R.-Q. Meng, X.-H. Meng, K.-S. Wang, H.-M. Xiao, et al., Accurate recognition of colorectal cancer with semi-supervised deep learning on pathological images, Nat. Commun., vol. 12, p. 6311, 2021.

Crossref Google Scholar
[6]

S. Sayed, R. Lukande, and K. A. Fleming, Providing pathology support in low-income countries, J. Glob. Oncol., vol. 1, no. 1, pp. 3–6, 2015.

Crossref Google Scholar
[7]

F. Yuan, Z. Zhang, and Z. Fang, An effective CNN and Transformer complementary network for medical image segmentation, Pattern Recognit., vol. 136, p. 109228, 2023.

Crossref Google Scholar
[8]

Y. Chen, Y. Zhou, G. Chen, Y. Guo, Y. Lv, M. Ma, Z. Pei, and Z. Sun, Segmentation of breast tubules in H&E images based on a DKS-DoubleU-net model, BioMed Res. Int., vol. 2022, p. 2961610, 2022.

Crossref Google Scholar
[9]
X. Li, A. Jiang, S. Wang, F. Li, and S. Yan, CTBP-Net: Lung nodule segmentation model based on the cross-transformer and bidirectional pyramid, Biomed. Signal Process. Contr., vol. 82, p. 104528, 2023.
Crossref
[10]

H. Xu, L. Liu, X. Lei, M. Mandal, and C. Lu, An unsupervised method for histological image segmentation based on tissue cluster level graph cut, Comput. Med. Imag. Graph., vol. 93, p. 101974, 2021.

Crossref Google Scholar
[11]

T. Mathew, B. Ajith, J. R. Kini, and J. Rajan, Deep learning-based automated mitosis detection in histopathology images for breast cancer grading, Int. J. Imag. Syst. Technol., vol. 32, no. 4, pp. 1192–1208, 2022.

Crossref Google Scholar
[12]

G. Xi, Q. Wang, H. Zhan, D. Kang, Y. Liu, T. Luo, M. Xu, Q. Kong, L. Zheng, G. Chen, et al., Automated classification of breast cancer histologic grade using multiphoton microscopy and generative adversarial networks, J. Phys. D: Appl. Phys., vol. 56, no. 1, p. 015401, 2023.

Crossref Google Scholar
[13]

Y. Wang, X. Lei, and Y. Pan, Predicting microbe-disease association based on heterogeneous network and global graph feature learning, Chin. J Electronics, vol. 31, no. 2, pp. 345–353, 2022.

Crossref Google Scholar
[14]

C. Fan, X. Lei, L. Guo, and A. Zhang, Predicting the associations between microbes and diseases by integrating multiple data sources and path-based HeteSim scores, Neurocomputing, vol. 323, pp. 76–85, 2019.

Crossref Google Scholar
[15]
H. Meng, X. Liu, J. Niu, Y. Wang, J. Liao, Q. Li, and C. Chen, DGANet: A dual global attention neural network for breast lesion detection in ultrasound images, Ultrasound Med. Biol., vol. 49, no. 1, pp. 31–44, 2023.
Crossref
[16]

S. Khaliliboroujeni, X. He, W. Jia, and S. Amirgholipour, End-to-end metastasis detection of breast cancer from histopathology whole slide images, Comput. Med. Imag. Graph., vol. 102, p. 102136, 2022.

Crossref Google Scholar
[17]

H. Mkindu, L. Wu, and Y. Zhao, Lung nodule detection in chest CT images based on vision transformer network with Bayesian optimization, Biomed. Signal Process. Contr., vol. 85, p. 104866, 2023.

Crossref Google Scholar
[18]

N. Zhang, Y.-X. Cai, Y.-Y. Wang, Y.-T. Tian, X.-L. Wang, and B. Badami, Skin cancer diagnosis based on optimized convolutional neural network, Artif. Intell. Med., vol. 102, p. 101756, 2020.

Crossref Google Scholar
[19]

W. Long, Y. Yang, and H.-B. Shen, ImPLoc: A multi-instance deep learning model for the prediction of protein subcellular localization based on immunohistochemistry images, Bioinformatics, vol. 36, no. 7, pp. 2244–2250, 2020.

Crossref Google Scholar
[20]

J.-X. Hu, Y. Yang, Y.-Y. Xu, and H.-B. Shen, Incorporating label correlations into deep neural networks to classify protein subcellular location patterns in immunohistochemistry images, Proteins Struct. Funct. Bioinform., vol. 90, no. 2, pp. 493–503, 2022.

Crossref Google Scholar
[21]

G. Shamai, Y. Binenbaum, R. Slossberg, I. Duek, Z. Gil, and R. Kimmel, Artificial intelligence algorithms to assess hormonal status from tissue microarrays in patients with breast cancer, JAMA Netw. Open, vol. 2, no. 7, p. e197700, 2019.

Crossref Google Scholar
[22]

A. Su, H. Lee, X. Tan, C. J. Suarez, N. Andor, Q. Nguyen, and H. P. Ji, A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images, NPJ Precis. Onc., vol. 6, no. 1, p. 14, 2022.

Crossref Google Scholar
[23]
L. Theelke, F. Wilm, C. Marzahl, C. A. Bertram, R. Klopfleisch, A. Maier, M. Aubreville, and K. Breininger, Iterative cross-scanner registration for whole slide images, in Proc. IEEE/CVF Int. Conf. Computer Vision Workshops (ICCVW), Montreal, Canada, 2021, pp. 582–590.
Crossref
[24]
A. Shafique, M. Babaie, M. Sajadi, A. Batten, S. Skdar, and H. R. Tizhoosh, Automatic multi-stain registration of whole slide images in histopathology, in Proc. 43rd Annual Int. Conf. IEEE Engineering in Medicine & Biology Society (EMBC), Virtual Event, 2021, pp. 3622–3625.
Crossref
[25]

L. Venet, S. Pati, M. D. Feldman, M. P. Nasrallah, P. Yushkevich, and S. Bakas, Accurate and robust alignment of differently stained histologic images based on greedy diffeomorphic registration, Appl. Sci., vol. 11, no. 4, p. 1892, 2021.

Crossref Google Scholar
[26]

J. Yao, X. Zhu, J. Jonnagaddala, N. Hawkins, and J. Huang, Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks, Med. Image Anal., vol. 65, p. 101789, 2020.

Crossref Google Scholar
[27]

M. Z. Hoque, A. Keskinarkaus, P. Nyberg, T. Mattila, and T. Seppänen, Whole slide image registration via multi-stained feature matching, Comput. Biol. Med., vol. 144, p. 105301, 2022.

Crossref Google Scholar
[28]
P. Liu, F. Wang, G. Teodoro, and J. Kong, Histopathology image registration by integrated texture and spatial proximity based landmark selection and modification, in Proc. IEEE 18th Int. Symp. on Biomedical Imaging (ISBI), Nice, France, 2021, pp. 1827–1830.
Crossref
[29]

L. Ge, X. Wei, Y. Hao, J. Luo, and Y. Xu, Unsupervised histological image registration using structural feature guided convolutional neural network, IEEE Trans. Med. Imag., vol. 41, no. 9, pp. 2414–2431, 2022.

Crossref Google Scholar
[30]
J. Lotz, N. Weiss, J. van der Laak, and S. Heldmann, Comparison of consecutive and re-stained sections for image registration in histopathology, arXiv preprint arXiv: 2106.13150, 2021.
[31]
S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, CBAM: Convolutional block attention module, in Proc. European Conference on Computer Vision (ECCV 2018), doi: 10.1007/978-3-030-01234-2_1.
Crossref
[32]

Y. Zhang, L. Luo, Q. Dou, and P.-A. Heng, Triplet attention and dual-pool contrastive learning for clinic-driven multi-label medical image classification, Med. Image Anal., vol. 86, p. 102772, 2023.

Crossref Google Scholar
[33]

J. Cheng, S. Tian, L. Yu, C. Gao, X. Kang, X. Ma, W. Wu, S. Liu, and H. Lu, ResGANet: Residual Group attention network for medical image classification and segmentation, Med. Image Anal., vol. 76, p. 102313, 2022.

Crossref Google Scholar
[34]

M. Karri, C. S. R. Annavarapu, and U. R. Acharya, Explainable multi-module semantic guided attention based network for medical image segmentation, Comput. Biol. Med., vol. 151, p. 106231, 2022.

Crossref Google Scholar
[35]
Q. Wan, Z. Huang, J. Lu, G. Yu, and L. Zhang, SeaFormer++: Squeeze-enhanced axial transformer for mobile visual recognition, arXiv preprint arXiv: 2301.13156, 2023.
[36]

R. Yan, Q. He, Y. Liu, P. Ye, L. Zhu, S. Shi, J. Gou, Y. He, T. Guan, and G. Zhou, Unpaired virtual histological staining using prior-guided generative adversarial networks, Comput. Med. Imag. Graph., vol. 105, p. 102185, 2023.

Crossref Google Scholar
[37]
S. Liu, C. Zhu, F. Xu, X. Jia, Z. Shi, and M. Jin, BCI: Breast cancer immunohistochemical image generation through pyramid Pix2pix, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), New Orleans, LA, USA, 2022, pp. 1814–1823.
Crossref
[38]
R. Zhang, Y. Cao, Y. Li, Z. Liu, J. Wang, J. He, C. Zhang, X. Sui, P. Zhang, L. Cui, et al., MVFStain: Multiple virtual functional stain histopathology images generation based on specific domain mapping, Med. Image Anal., vol. 80, p. 102520, 2022.
Crossref
[39]

P. Gamble, R. Jaroensri, H. Wang, F. Tan, M. Moran, T. Brown, I. Flament-Auvigne, E. A. Rakha, M. Toss, D. J. Dabbs, et al., Determining breast cancer biomarker status and associated morphological features using deep learning, Commun. Med., vol. 1, no. 1, p. 14, 2021.

Crossref Google Scholar
[40]
R. R. Rawat, I. Ortega, P. Roy, F. Sha, D. Shibata, D. Ruderman, and D. B. Agus, Deep learned tissue “fingerprints” classify breast cancers by ER/PR/Her2 status from H&E images, Scientific Reports, vol. 10, p. 7275, 2020.
Crossref
[41]
J. Chen, Y. Lu, Q. Yu, X. Luo, E. Adeli, Y. Wang, L. Lu, A. L. Yuille, and Y. Zhou, TransUNet: Transformers make strong encoders for medical image segmentation, arXiv preprint arXiv:2102.04306, 2021.
[42]
Y. Zhang, H. Liu, and Q. Hu, Transfuse: Fusing transformers and CNNs for medical image segmentation, in Proc. 24th Int. Conf. Medical Image Computing and Computer Assisted Intervention – MICCAI, Strasbourg, France, 2021, pp. 14 –24.
Crossref
[43]
Y. Chen, X. Dai, D. Chen, M. Liu, X. Dong, L. Yuan, and Z. Liu, Mobile-former: Bridging MobileNet and transformer, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 5270–5279.
Crossref
[44]
Z. Peng, W. Huang, S. Gu, L. Xie, Y. Wang, J. Jiao, and Q. Ye, Conformer: Local features coupling global representations for visual recognition, in Proc. IEEE/CVF Int. Conf. Computer Vision (ICCV), Montreal, Canada, 2021, pp. 357–366.
Crossref
[45]
Z. Zhang, G. Sun, K. Zheng, J.-K. Yang, X.-R. Zhu, and Y. Li, TC-Net: A joint learning framework based on CNN and vision transformer for multi-lesion medical images segmentation, Comput. Biol. Med., vol. 161, p. 106967, 2023.
Crossref
[46]

A. Vahadane, T. Peng, A. Sethi, S. Albarqouni, L. Wang, M. Baust, K. Steiger, A. M. Schlitter, I. Esposito, and N. Navab, Structure-preserving color normalization and sparse stain separation for histological images, IEEE Trans. Med. Imag., vol. 35, no. 8, p. 1962–1971, 2016.

Crossref Google Scholar
[47]

A. C. Ruifrok and D. A. Johnston, Quantification of histochemical staining by color deconvolution, Anal. Quant. Cytol. Histol., vol. 23, no. 4, pp. 291–299, 2001.

Google Scholar
[48]

F. Shamshad, S. Khan, S. W. Zamir, M. H. Khan, M. Hayat, F. S. Khan, and H. Fu, Transformers in medical imaging: A survey, Med. Image Anal., vol. 88, p. 102802, 2023.

Crossref Google Scholar
[49]
Q. He, Q. Yang, and M. Xie, HCTNet: A hybrid CNN-transformer network for breast ultrasound image segmentation, Comput. Biol. Med., vol. 155, p. 106629, 2023.
Crossref
[50]
J. M. J. Valanarasu, P. Oza, I. Hacihaliloglu, and V. M. Patel, Medical transformer: Gated axial-attention for medical image segmentation, in Proc. 24th Int. Conf. Medical Image Computing and Computer Assisted Intervention – MICCAI, Strasbourg, France, 2021, pp. 36–46.
Crossref
[51]
Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, A ConvNet for the 2020s, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 11966–11976.
Crossref
[52]
S. Ioffe, Batch renormalization: Towards reducing minibatch dependence in batch-normalized models, arXiv preprint arXiv: 1702.03275, 2017.
[53]
J. L. Ba, J. R. Kiros, and G. E. Hinton, Layer normalization, arXiv preprint arXiv: 1607.06450, 2016.
[54]
T. Qaiser, A. Mukherjee, C. R. Pb, S. D. Munugoti, V. Tallam, T. Pitkäaho, T. Lehtimäki, T. Naughton, M. Berseth, A. Pedraza, et al., HER2 challenge contest: A detailed assessment of automated HER2 scoring algorithms in whole slide images of breast cancer tissues, Histopathology, vol. 72, no. 2, pp. 227–238, 2018.
Crossref
[55]

Y. Li, T. Yao, Y. Pan, and T. Mei, Contextual transformer networks for visual recognition, IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 2, pp. 1489–1500, 2023.

Crossref Google Scholar
[56]
Y. Liu, Z. Shao, and N. Hoffmann, Global attention mechanism: Retain information to enhance channel-spatial interactions, arXiv preprint arXiv: 2112.05561, 2021.
[57]
K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 770–778.
Crossref
[58]
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, MobileNetV2: Inverted residuals and linear bottlenecks, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 4510–4520.
Crossref
[59]
K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556, 2014.
[60]

B. Fu, M. Zhang, J. He, Y. Cao, Y. Guo, and R. Wang, StoHisNet: A hybrid multi-classification model with CNN and Transformer for gastric pathology images, Comput. Meth. Programs Biomed., vol. 221, p. 106924, 2022.

Crossref Google Scholar
[61]
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, Swin transformer: Hierarchical vision transformer using shifted windows, in Proc. IEEE/CVF Int. Conf. Computer Vision (ICCV), Montreal, Canada, 2021, pp. 9992–10002.
Crossref
[62]

Y. Liu, X. Li, A. Zheng, X. Zhu, S. Liu, M. Hu, Q. Luo, H. Liao, M. Liu, Y. He, et al., Predict Ki-67 positive cells in H&E-stained images using deep learning independently from IHC-stained images, Front. Mol. Biosci., vol. 7, p. 183, 2020.

Crossref Google Scholar
[63]

S. H. Lee, I. H. Song, and H.-J. Jang, Feasibility of deep learning-based fully automated classification of microsatellite instability in tissue slides of colorectal cancer, Int. J. Cancer, vol. 149, no. 3, pp. 728–740, 2021.

Crossref Google Scholar
[64]

R. J. Chen, T. Ding, M. Y. Lu, D. F. K. Williamson, G. Jaume, A. H. Song, B. Chen, A. Zhang, D. Shao, M. Shaban, et al., Towards a general-purpose foundation model for computational pathology, Nat. Med., vol. 30, no. 3, pp. 850–862, 2024.

Crossref Google Scholar
[65]
D. C. Bui, B. Song, K. Kim, and J. T. Kwak, DAX-Net: A dual-branch dual-task adaptive cross-weight feature fusion network for robust multi-class cancer classification in pathology images, Comput. Meth. Programs Biomed., vol. 248, p. 108112, 2024.
Crossref
[66]

X. Huo, G. Sun, S. Tian, Y. Wang, L. Yu, J. Long, W. Zhang, and A. Li, HiFuse: Hierarchical multi-scale feature fusion network for medical image classification, Biomed. Signal Process. Contr., vol. 87, p. 105534, 2024.

Crossref Google Scholar
[67]

L. Van Der Maaten and G. Hinton, Visualizing data using t-SNE, J. Mach. Learn. Res., vol. 9, pp. 2579–2625, 2008.

Google Scholar
[68]
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, Grad-CAM: Visual explanations from deep networks via gradient-based localization, in Proc. IEEE Int. Conf. Computer Vision (ICCV), Venice, Italy, 2017, pp. 618–626.
Crossref
Big Data Mining and Analytics
Volume 7 Issue 4,
December 2024
Pages 1212-1236
DOI: 10.26599/BDMA.2024.9020057
Cite this article:
Chen Y, Chen G, Shi G, et al. SeaConvNeXt: A Lightweight Two-Branch Network Architecture for Efficient Prediction of Specific IHC Proteins and Antigens on Hematoxylin and Eosin (H&E) Images. Big Data Mining and Analytics, 2024, 7(4): 1212-1236. https://doi.org/10.26599/BDMA.2024.9020057
Metrics & Citations  
Article History
Copyright
Rights and Permissions
Return