LEAKAGE-CONTROLLED EVALUATION OF MACHINE LEARNING FOR HEART DISEASE DIAGNOSIS AND HEART FAILURE MORTALITY PREDICTION

Abstract

Reported machine-learning performance on small public heart datasets varies, and incomplete protocols limit reproducibility. This verification study assessed reported accuracy under a leakage-controlled protocol rather than attempted to improve on the base study. We analyzed 302 Cleveland diagnostic records after one duplicate was removed and 299 Faisalabad heart-failure records in three configurations: Cleveland diagnosis, Faisalabad mortality with follow-up time, and Faisalabad mortality without follow-up time. Nine classifiers and four variants were assessed by hold-out evaluation and repeated nested cross-validation (CV), with ten outer folds and paired tests. The area under the receiver operating characteristic curve (ROC AUC, hereafter AUC) was a tuning target. Shapley Additive explanations (SHAP) and the rank-sum selector described attribution and feature rankings. In Cleveland, default_all random forest accuracy was 0.902 (55 of 61 hold-out records), matching the base-study figure numerically (not an exact reproduction), while its nested accuracy was 0.836 and nested AUC was 0.909. The tuned random forest reached 0.918 hold-out accuracy and 0.846 nested accuracy. Across Faisalabad default models, mean nested AUC decreased from 0.828 with follow-up time to 0.710 without time. For tuned models, the corresponding means were 0.869 and 0.751. Follow-up time accounted for 39.9 percent of random-forest SHAP attribution when included. These results support excluding follow-up duration from baseline mortality prediction. Comparison with the base study is indirect because its split, seed, and placement of feature selection are not stated. Small samples, one hold-out split, correlated outer folds, and lack of external validation limit inference.