EXPLANATION STABILITY IN CROSS-PROJECT DEFECT PREDICTION: RETRAINING, EXPLAINER AGREEMENT, AND PERTURBATION ROBUSTNESS

Abstract

Explainable artificial intelligence (XAI) techniques such as SHAP and LIME are increasingly used to interpret software defect prediction models, but the attributions they produce are often treated as fixed properties of a model rather than as outputs whose reproducibility must itself be checked. In cross-project defect prediction (CPDP), where a model trained on some projects is applied to another, this concern has several dimensions that prior work has not examined within a single protocol. We empirically evaluate four of them: retraining stability, cross-project consistency, SHAP–LIME agreement, and robustness to prediction-preserving input perturbations. Using the five AEEEM projects (5,371 modules, 61 numeric metrics) under leave-one-project-out validation, we fitted Random Forest and HistGradientBoosting models with ten random seeds per target (100 fits), and analysed 200 matched target modules with a seed-42 Random Forest. Global SHAP rankings were highly stable across Random Forest retraining (mean Spearman ρ = 0.9806) but less consistent across target projects (ρ = 0.8457), and top-10 feature-set overlap fell from a Jaccard index of 0.8361 to 0.4457. For identical modules, SHAP and LIME shared on average 1.90 of their top five metrics (Jaccard@5 = 0.2512). Under perturbations of 1% to 5% of a source-feature standard deviation, the share of prediction-stable perturbations fell from 89.2% to 66.2%, and among them SHAP top-5 similarity declined from 0.7773 to 0.6644. Within this benchmark, explanation reproducibility is multidimensional: highly repeatable global explanations can coexist with limited cross-project portability, limited cross-explainer agreement, and gradual degradation under output-preserving perturbations. Explanation reliability should therefore be reported alongside predictive performance.