A HYBRID DEEP LEARNING FRAMEWORK FOR BREAST CANCER DETECTION IN MAMMOGRAPHY IMAGES USING IMBALANCE-AWARE TRAINING

Abstract

Breast cancer is one of the most prevalent malignancies worldwide, and early detection through mammography significantly improves survival rates. This paper proposes a hybrid deep learning framework for automated breast cancer detection using the RSNA Breast Cancer Detection dataset comprising 54,706 high-resolution 512×512 mammography images. The framework integrates a Custom Convolutional Neural Network (CCNN) with a fine-tuned VGG-16 transfer learning model to capture both low-level spatial features and high-level semantic representations.

To address severe class imbalance, the proposed approach employs targeted data augmentation on cancerous samples, controlled undersampling of non-cancerous samples, class weighting, and focal loss. Image quality is enhanced using Contrast Limited Adaptive Histogram Equalization (CLAHE), followed by multi-channel input construction and normalization. A patient-wise splitting strategy is adopted to prevent data leakage, dividing the dataset into 70% training, 15% validation, and 15% testing subsets. Training stability is ensured through early stopping, adaptive learning rate scheduling, and batch normalization.

Experimental results on the unseen test set demonstrate outstanding performance, achieving 99.85% accuracy, 99.80% precision, 99.95% recall, and 99.87% F1-score. These results significantly outperform baseline CNN architectures and the reference concatenation model (92% accuracy). The proposed framework offers a robust and clinically promising solution for mammography-based breast cancer detection.