Skip to main content

Advertisement

Springer Nature Link
Log in

Classifying the molecular subtype of breast cancer using vision transformer and convolutional neural network features

  • Research
  • Published:
Breast Cancer Research and Treatment Aims and scope Submit manuscript

Abstract

Purpose

Identification of the molecular subtypes in breast cancer allows to optimize treatment strategies, but usually requires invasive needle biopsy. Recently, non-invasive imaging has emerged as promising means to classify them. Magnetic resonance imaging is often used for this purpose because it is three-dimensional and highly informative. Instead, only a few reports have documented the use of mammograms. Given that mammography is the first choice for breast cancer screening, using it to classify molecular subtypes would allow for early intervention on a much wider scale. Here, we aimed to evaluate the effectiveness of combining global and local mammographic features by using Vision Transformer (ViT) and Convolutional Neural Network (CNN) to classify molecular subtypes in breast cancer.

Methods

The feature values for binary classification were calculated using the ViT and EfficientnetV2 feature extractors, followed by dimensional compression via principal component analysis. LightGBM was used to perform binary classification of each molecular subtype: triple-negative, HER2-enriched, luminal A, and luminal B.

Results

The combination of ViT and CNN achieved higher accuracy than ViT or CNN alone. The sensitivity for triple-negative subtypes was very high (0.900, with F-value = 0.818); whereas F-value and sensitivity were 0.720 and 0.750 for HER2-enriched, 0.765 and 0.867 for luminal A, and 0.614 and 0.711 for luminal B subtypes, respectively.

Conclusion

Features obtained from mammograms by combining ViT and CNN allow the classification of molecular subtypes with high accuracy. This approach could streamline early treatment workflows and triage, especially for poor prognosis subtypes such as triple-negative breast cancer.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Canada)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

The datasets analyzed during the current study are available in The Chinese Mammography Database (CMMD) repository, The Cancer Imaging Archive (https://doi.org/https://doi.org/10.7937/tcia.eqde-4b16).

Abbreviations

AI:

Artificial intelligence

CC:

Craniocaudal

CNN:

Convolutional neural network

CMMD:

Chinese mammography database

CESM:

Contrast-enhanced spectral mammography

MLO:

Mediolateral oblique

MRI:

Magnetic resonance imaging

PCA:

Principal component analysis

ViT:

Vision transformer

References

  1. Curigliano G, Burstein HJ, Gnant M, Loibl S, Cameron D, Regan MM et al (2023) Understanding breast cancer complexity to improve patient outcomes: the st gallen international consensus conference for the primary therapy of individuals with early breast cancer 2023. Ann Oncol 34:970–986. https://doi.org/10.1016/j.annonc.2023.08.017

    Article  CAS  PubMed  Google Scholar 

  2. Amoroso N, Pomarico D, Fanizzi A, Didonna V, Giotta F, La Forgia D et al (2021) A roadmap towards breast cancer therapies supported by explainable artificial intelligence. Appl Sci 11:4881. https://doi.org/10.3390/app11114881

    Article  CAS  Google Scholar 

  3. Perou CM, Sørlie T, Eisen MB, van de Rijn M, Jeffrey SS, Rees CA et al (2000) Molecular portraits of human breast tumours. Nature 406:747–752. https://doi.org/10.1038/35021093

    Article  CAS  PubMed  Google Scholar 

  4. Lambin P, Rios-Velazquez E, Leijenaar R, Carvalho S, van Stiphout RGPM, Granton P et al (2012) Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer 48:441–446. https://doi.org/10.1016/j.ejca.2011.11.036

    Article  PubMed  PubMed Central  Google Scholar 

  5. Demircioglu A, Grueneisen J, Ingenwerth M, Hoffmann O, Pinker-Domenig K, Morris E et al (2020) A rapid volume of interest-based approach of radiomics analysis of breast MRI for tumor decoding and phenotyping of breast cancer. PLoS ONE 15:e0234871. https://doi.org/10.1371/journal.pone.0234871

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Fan M, Liu Z, Xu M, Wang S, Zeng T, Gao X, Li L (2020) Generative adversarial network-based super-resolution of diffusion-weighted imaging: application to tumour radiomics in breast cancer. NMR Biomed 33:e4345. https://doi.org/10.1002/nbm.4345

    Article  CAS  PubMed  Google Scholar 

  7. Leithner D, Mayerhoefer ME, Martinez DF, Jochelson MS, Morris EA, Thakur SB, Pinker K (2020) Non-invasive assessment of breast cancer molecular subtypes with multiparametric magnetic resonance imaging radiomics. J Clin Med 9:1853. https://doi.org/10.3390/jcm9061853

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Ma M, Gan L, Jiang Y, Qin N, Li C, Zhang Y, Wang X (2021) Radiomics analysis based on automatic image segmentation of DCE-MRI for predicting triple-negative and nontriple-negative breast cancer. Comput Math Methods Med 2021:2140465. https://doi.org/10.1155/2021/2140465

    Article  PubMed  PubMed Central  Google Scholar 

  9. Yue WY, Zhang HT, Gao S, Li G, Sun ZY, Tang Z et al (2023) Predicting breast cancer subtypes using magnetic resonance imaging based radiomics with automatic segmentation. J Comput Assist Tomogr 47:729–737. https://doi.org/10.1097/RCT.0000000000001474

    Article  PubMed  PubMed Central  Google Scholar 

  10. Huang G, Du S, Gao S, Guo L, Zhao R, Bian X et al (2024) Molecular subtypes of breast cancer identified by dynamically enhanced MRI radiomics: the delayed phase cannot be ignored. Insights Imag 15:127. https://doi.org/10.1186/s13244-024-01713-9

    Article  Google Scholar 

  11. Ha R, Mutasa S, Karcich J, Gupta N, Pascual Van Sant E, Nemer J et al (2019) Predicting breast cancer molecular subtype with MRI dataset utilizing convolutional neural network algorithm. J Digit Imaging 32:276–282. https://doi.org/10.1007/s10278-019-00179-2

    Article  PubMed  PubMed Central  Google Scholar 

  12. Sun R, Meng Z, Hou X, Chen Y, Yang Y, Huang G, Nie S (2021) Prediction of breast cancer molecular subtypes using DCE-MRI based on CNNs combined with ensemble learning. Phys Med Biol. https://doi.org/10.1088/1361-6560/ac195a

    Article  PubMed  Google Scholar 

  13. Zhang Y, Chen JH, Lin Y, Chan S, Zhou J, Chow D et al (2021) Prediction of breast cancer molecular subtypes on DCE-MRI using convolutional neural network with transfer learning between two centers. Eur Radiol 31:2559–2567. https://doi.org/10.1007/s00330-020-07274-x

    Article  PubMed  Google Scholar 

  14. Li W, Wang S, Xie W, Feng C (2023) A quantitative heterogeneity analysis approach to molecular subtype recognition of breast cancer in dynamic contrast-enhanced magnetic imaging images from radiomics data. Quant Imaging Med Surg 13:4429–4446. https://doi.org/10.21037/qims-22-1230

    Article  PubMed  PubMed Central  Google Scholar 

  15. Xie X, Zhou H, Ma M, Nie J, Gao W, Zhong J et al (2024) A deep learning model for predicting molecular subtype of breast cancer by fusing multiple sequences of DCE-MRI from two institutes. Acad Radiol. https://doi.org/10.1016/j.acra.2024.03.002

    Article  PubMed  Google Scholar 

  16. Petrillo A, Fusco R, Barretta ML, Granata V, Mattace Raso M, Porto A et al (2023) Radiomics and artificial intelligence analysis by T2-weighted imaging and dynamic contrast-enhanced magnetic resonance imaging to predict breast cancer histological outcome. Radiol Med 128:1347–1371. https://doi.org/10.1007/s11547-023-01718-2

    Article  PubMed  Google Scholar 

  17. Gu J, Jiang T (2022) Ultrasound radiomics in personalized breast management: current status and future prospects. Front Oncol 12:963612. https://doi.org/10.3389/fonc.2022.963612

    Article  PubMed  PubMed Central  Google Scholar 

  18. Jiang M, Zhang D, Tang SC, Luo XM, Chuan ZR, Lv WZ et al (2021) Deep learning with convolutional neural network in the assessment of breast cancer molecular subtypes based on US images: a multicenter retrospective study. Eur Radiol 31:3673–3682. https://doi.org/10.1007/s00330-020-07544-8

    Article  PubMed  Google Scholar 

  19. La Forgia D, Fanizzi A, Campobasso F, Bellotti R, Didonna V, Lorusso V et al (2020) Radiomic analysis in contrast-enhanced spectral mammography for predicting breast cancer histological outcome. Diagnostics (Basel) 10:708. https://doi.org/10.3390/diagnostics10090708

    Article  PubMed  Google Scholar 

  20. Petrillo A, Fusco R, Di Bernardo E, Petrosino T, Barretta ML, Porto A et al (2022) Prediction of breast cancer histological outcome by radiomics and artificial intelligence analysis in contrast-enhanced mammography. Cancers (Basel) 14:2132. https://doi.org/10.3390/cancers14092132

    Article  CAS  PubMed  Google Scholar 

  21. Ma W, Zhao Y, Ji Y, Guo X, Jian X, Liu P, Wu S (2019) Breast cancer molecular subtype prediction by mammographic radiomic features. Acad Radiol 26:196–201. https://doi.org/10.1016/j.acra.2018.01.023

    Article  PubMed  Google Scholar 

  22. Ueda D, Yamamoto A, Takashima T, Onoda N, Noda S, Kashiwagi S et al (2021) Training, validation, and test of deep learning models for classification of receptor expressions in breast cancers from mammograms. JCO Precis Oncol 5:543–551. https://doi.org/10.1200/PO.20.00176

    Article  PubMed  Google Scholar 

  23. Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M et al (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88. https://doi.org/10.1016/j.media.2017.07.005

    Article  PubMed  Google Scholar 

  24. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, et al An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929

  25. Ali H, Mohsen F, Shah Z (2023) Improving diagnosis and prognosis of lung cancer using vision transformers: a scoping review. BMC Med Imaging 23:129. https://doi.org/10.1186/s12880-023-01098-z

    Article  PubMed  PubMed Central  Google Scholar 

  26. Gheflati B, Rivaz H. Vision transformers for classification of breast ultrasound images (2022) Annu Int Conf IEEE Eng Med Biol Soc 480–483. https://doi.org/10.1109/EMBC48229.2022.9871809.

  27. Chen X, Zhang K, Abdoli N, Gilley PW, Wang X, Liu H et al (2022) Transformers improve breast cancer diagnosis from unregistered multi-view mammograms. Diagnostics (Basel) 12:1549. https://doi.org/10.3390/diagnostics12071549

    Article  CAS  PubMed  Google Scholar 

  28. Lian J, Deng J, Hui ES, Koohi-Moghadam M, She Y, Chen C et al (2022) Early stage NSCLS patients’ prognostic prediction with multi-information using transformer and graph neural network model. Elife 11:e80547. https://doi.org/10.7554/eLife.80547

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  29. Ayana G, Dese K, Dereje Y, Kebede Y, Barki H, Amdissa D et al (2023) Vision-transformer-based transfer learning for mammogram classification. Diagnostics (Basel) 13:178. https://doi.org/10.3390/diagnostics13020178

    Article  PubMed  Google Scholar 

  30. Fanizzi A, Fadda F, Comes MC, Bove S, Catino A, Di Benedetto E et al (2023) Comparison between vision transformers and convolutional neural networks to predict non-small lung cancer recurrence. Sci Rep 13:20605. https://doi.org/10.1038/s41598-023-48004-9

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Qu X, Lu H, Tang W, Wang S, Zheng D, Hou Y, Jiang J (2022) A VGG attention vision transformer network for benign and malignant classification of breast ultrasound images. Med Phys 49:5787–5798. https://doi.org/10.1002/mp.15852

    Article  CAS  PubMed  Google Scholar 

  32. Tagnamas J, Ramadan H, Yahyaouy A, Tairi H (2024) Multi-task approach based on combined CNN-transformer for efficient segmentation and classification of breast tumors in ultrasound images. Vis Comput Ind Biomed Art 7:2. https://doi.org/10.1186/s42492-024-00155-w [published correction appears in Vis Comput Ind Biomed Art 2024 7:5. https://doi.org/10.1186/s42492-024-00156-910.1186/s42492-024-00156-9]

  33. Xiong Y, Du B, Xu Y, Deng J, She Y, Chen C (2022) Pulmonary nodule classification with multi-view convolutional Vision Transformer. Int Joint Conference Neural Networks (IJCNN) 2022:1–7. https://doi.org/10.1109/IJCNN55064.2022.9892716

    Article  Google Scholar 

  34. Cui C, Li L, Cai H, Fan Z, Zhang L, Dan T et al (2021) The chinese mammography database (CMMD): an online mammography database with biopsy confirmed types for machine diagnosis of breast. Cancer Imaging Arch. https://doi.org/10.7937/tcia.eqde-4b16

    Article  Google Scholar 

  35. Cai H, Wang J, Dan T, Li J, Fan Z, Yi W et al (2023) An online mammography database with biopsy confirmed types. Sci Data 10:123. https://doi.org/10.1038/s41597-023-02025-1

    Article  PubMed  PubMed Central  Google Scholar 

  36. https://github.com/huggingface/pytorch-image-models

  37. https://huggingface.co/timm/vit_small_patch8_224.dino

  38. https://huggingface.co/timm/tf_efficientnetv2_m.in21k_ft_in1k

  39. Mingxing Tan, Quoc v. Le EfficientNetV2: Smaller models and faster training. arXiv:2104.00298

  40. Akiba T, Sano S, Yanase T, Ohta T, Koyama M (2019) Optuna: A next-generation hyperparameter optimization framework. https://doi.org/10.1145/3292500.3330701, arXiv:1907.10902

  41. American College of Radiology (2013) Breast Imaging Reporting and Data System: BI-RADS atlas, 5th edn. American College of Radiology, Reston

  42. Ko ES, Lee BH, Kim HA, Noh WC, Kim MS, Lee SA (2010) Triple-negative breast cancer: correlation between imaging and pathological findings. Eur Radiol 20:1111–1117. https://doi.org/10.1007/s00330-009-1656-3

    Article  PubMed  Google Scholar 

  43. Tamaki K, Ishida T, Miyashita M, Amari M, Ohuchi N, Tamaki N, Sasano H (2011) Correlation between mammographic findings and corresponding histopathology: potential predictors for biological characteristics of breast diseases. Cancer Sci 102:2179–2185. https://doi.org/10.1111/j.1349-7006.2011.02088.x

    Article  CAS  PubMed  Google Scholar 

  44. Evans AJ, Pinder SE, James JJ, Ellis IO, Cornford E (2006) Is mammographic spiculation an independent, good prognostic factor in screening-detected invasive breast cancer? AJR Am J Roentgenol 187:1377–1380. https://doi.org/10.2214/AJR.05.0725

    Article  PubMed  Google Scholar 

  45. Wang Y, Ikeda DM, Narasimhan B, Longacre TA, Bleicher RJ, Pal S et al (2008) Estrogen receptor-negative invasive breast cancer: Imaging features of tumors with and without human epidermal growth factor receptor type 2 overexpression. Radiology 246:367–375. https://doi.org/10.1148/radiol.2462070169

    Article  PubMed  Google Scholar 

  46. Gastounioti A, Desai S, Ahluwalia VS, Conant EF, Kontos D (2022) Artificial intelligence in mammographic phenotyping of breast cancer risk: a narrative review. Breast Cancer Res 24:14. https://doi.org/10.1186/s13058-022-01509-z

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  47. Li C, Lu N, He Z, Tan Y, Liu Y, Chen Y et al (2022) A noninvasive tool based on magnetic resonance imaging radiomics for the preoperative prediction of pathological complete response to neoadjuvant chemotherapy in breast cancer. Ann Surg Oncol 29:7685–7693. https://doi.org/10.1245/s10434-022-12034-w

    Article  PubMed  PubMed Central  Google Scholar 

  48. Cheng Y, Xu S, Wang H, Wang X, Niu S, Luo Y, Zhao N (2022) Intra- and peri-tumoral radiomics for predicting the sentinel lymph node metastasis in breast cancer based on preoperative mammography and MRI. Front Oncol 12:1047572. https://doi.org/10.3389/fonc.2022.1047572

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We would like to thank Yuta Hirono from Niigata University of Health and Welfare for useful discussions. We would like to thank Editage (www.editage.jp) for English language editing. This work was supported by the JSPS Grant-in-Aid for Scientific Research (C) (Grant Number JP23K10899).

Funding

This work was supported by the JSPS Grant-in-Aid for Scientific Research (C) (Grant Number JP23K10899).

Author information

Authors and Affiliations

  1. Department of Radiological Technology, Faculty of Medical Technology, Niigata University of Health and Welfare, 1398 Shimamichou, Kita-Ku, Niigata, Japan

    Chiharu Kai, Akifumi Yoshida, Naoki Kodama & Satoshi Kasai

  2. Major in Health and Welfare, Graduate School of Niigata, University of Health and Welfare, Niigata, Japan

    Chiharu Kai & Ikumi Sato

  3. The Asahi Shimbun Company, Tokyo, Japan

    Hideaki Tamori

  4. Ohtsuka Breastcare Clinic, Tokyo, Japan

    Tsunehiro Ohtsuka & Miyako Nara

  5. Department of Nursing, Faculty of Nursing, Niigata University of Health and Welfare, Niigata City, Niigata, Japan

    Ikumi Sato

  6. KONICA MINOLTA, INC, Tokyo, Japan

    Hitoshi Futamura

Authors
  1. Chiharu Kai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hideaki Tamori
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tsunehiro Ohtsuka
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Miyako Nara
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Akifumi Yoshida
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ikumi Sato
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Hitoshi Futamura
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Naoki Kodama
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Satoshi Kasai
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the conception and design of the study. Chiharu Kai, Hideaki Tamori, and Satoshi Kasai performed formal analyses. All authors have read and approved the final manuscript.

Corresponding author

Correspondence to Satoshi Kasai.

Ethics declarations

Competing Interests

Chiharu Kai and Satoshi Kasai were employed and received salaries from KONICA MINOLTA, INC. Satoshi Kasai received a research grant and consulting fee and has stocks from KONICA MINOLTA, INC. Tsunehiro Ohtsuka received research funding from KONICA MINOLTA, INC. Hideaki Tamori is affiliated with The Asahi Shimbun Company. Hitoshi Futamura is affiliated with KONICA MINOLTA, INC. The remaining authors declare that this study was conducted in the absence of commercial or financial relationships that could be construed as potential conflicts of interest.

Ethical approval

This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by The Institutional Review Board of Niigata University of Health and Welfare (Approval No. 19313–240613).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kai, C., Tamori, H., Ohtsuka, T. et al. Classifying the molecular subtype of breast cancer using vision transformer and convolutional neural network features. Breast Cancer Res Treat (2025). https://doi.org/10.1007/s10549-025-07614-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10549-025-07614-9

Keywords

Search

Navigation