A UNIFIED STATISTICAL–MACHINE LEARNING FRAMEWORK FOR CAUSAL DISCOVERY, EXPLAINABILITY, AND UNCERTAINTY QUANTIFICATION IN HIGH-DIMENSIONAL DATA

Abstract

High-dimensional data present substantial challenges for reliable prediction, interpretable modelling, causal discovery, and uncertainty quantification. Although statistical learning and explainable artificial intelligence have advanced considerably, these components are often investigated independently, limiting their combined effectiveness in scientific discovery and evidence-based decision-making. This study proposes a unified statistical–machine learning framework that integrates data preprocessing, dimensionality assessment, regularized prediction, model explainability, causal discovery, and uncertainty evaluation within a coherent analytical workflow. The framework is evaluated using a dataset comprising 5,000 observations and 500 numerical features, with Elastic Net regression employed to examine predictive performance and identify informative feature relationships. Model performance is assessed using the coefficient of determination (R²), root mean squared error (RMSE), and mean absolute error (MAE), while coefficient interpretation, permutation-based feature importance, SHAP analysis, resampling, and sensitivity assessment provide complementary approaches for examining model behaviour and result stability. The predictive model achieved an (R²) of 0.795, an RMSE of 0.446, and an MAE of 0.360, indicating that it explained 79.5% of the variation in the prediction target on the held-out evaluation data. These findings demonstrate the predictive potential of regularized machine learning in high-dimensional settings and establish a quantitative basis for integrating predictive performance with interpretability and uncertainty assessment. The proposed framework also emphasizes a fundamental distinction between predictive associations and causal relationships: feature importance and predictive accuracy alone cannot establish causation. Reliable causal discovery requires explicit structural assumptions, appropriate treatment of potential confounding, and systematic validation of the inferred relationships. By bringing statistical modelling, explainable artificial intelligence, causal reasoning, and uncertainty assessment together, this study provides a methodological foundation for more transparent, robust, and scientifically defensible analysis of complex data. The framework has potential applications in healthcare, financial risk assessment, environmental modelling, and other domains where accurate predictions must be accompanied by interpretable evidence and carefully evaluated uncertainty.