MACHINE LEARNING AND DEEP LEARNING APPROACHES FOR STRENGTHENING CYBER SECURITY IN INTRUSION DETECTION SYSTEM

Abstract

The rapid evolution of cyber threats, including advanced persistent attacks (APTs), polymorphic malware, ransomware, and zero-day exploits, has significantly increased the complexity and frequency of network intrusions, thereby necessitating the development of robust and intelligent intrusion detection systems (IDS). Traditional IDS techniques, primarily based on signature matching and statistical anomaly detection, are increasingly inadequate in modern cybersecurity environments due to their dependence on predefined attack patterns and limited capability to generalize across unseen or evolving threats. These systems often suffer from high false alarm rates, poor detection of novel attacks, and inefficiencies when deployed in large-scale, dynamic network infrastructures. To address these limitations, this study proposes a comprehensive and unified framework that integrates both machine learning (ML) and deep learning (DL) techniques for enhancing IDS performance. The framework is designed to systematically evaluate and compare classical ML models such as Support Vector Machines (SVM), Random Forest (RF), and Extreme Gradient Boosting (XG Boost) with advanced DL architectures, including Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM) networks, Transformer-based models, Autoencoders, and Graph Neural Networks (GNN). These models are assessed using widely recognized benchmark datasets, namely NSL-KDD, CICIDS2017, and UNSW-NB15, which collectively provide diverse and realistic representations of network traffic and attack scenarios The proposed methodology incorporates a multi-stage pipeline that includes data preprocessing (handling missing values, normalization, and encoding), feature engineering and selection (using statistical and model-based approaches), class imbalance handling (through techniques such as Synthetic Minority Over-sampling Technique (SMOTE) and cost-sensitive learning), and rigorous hyperparameter optimization using grid search and cross-validation strategies. This structured approach ensures reproducibility, robustness, and fair comparison across different models and datasets. Experimental evaluations reveal that deep learning models, particularly CNN and Transformer architectures, outperform traditional methods in capturing complex spatial and temporal patterns in network traffic data. These models achieve detection accuracies of up to 97.5%, along with significantly reduced false alarm rates as low as 1.4%, demonstrating their effectiveness in identifying both known and previously unseen attack patterns. Meanwhile, tree-based ML models such as XG Boost remain competitive, especially in scenarios involving structured tabular data, offering advantages in terms of interpretability and computational efficiency