SELECTIVE PREDICTION UNDER CROSS-PROJECT SHIFT: ABSTENTION RISK AND DEFECT-ENRICHED REVIEW QUEUES

Authors

  • Muhammad Arif Shah

Abstract

Cross-project defect prediction (CPDP) models are applied to projects whose data they were not trained on, yet they are normally evaluated as if every target module must receive an automated decision. Prior work has explored reject-option classification for software defect prediction, but there is limited evidence on how abstention behaves under cross-project shift, which reliability signals support it, and what happens to the modules a model declines to classify. We evaluate selective prediction with human fallback on two public benchmark families, AEEEM (five projects, 61 metrics) and NASA PROMISE (five projects, 20 metrics), under leave-one-project-out validation. Logistic Regression, Random Forest (RF) and HistGradientBoosting provide predictive baselines; for RF we compare four reliability signals for abstention: probability confidence, source distance, RF tree disagreement and RF seed disagreement. Each signal is assessed with risk-coverage curves, the area under the risk-coverage curve (AURC), matched random rejection and an oracle reference. An audit showed substantial within-project duplication and exact cross-project feature-vector overlap in the raw NASA data, so final NASA results use deduplicated and decontaminated data. In that analysis, deferring 20% of target modules reduced mean selective risk from 0.1886 under random rejection to 0.1418 with RF tree disagreement, the most consistently competitive signal (mean rank 1.6, best on 3 of 5 targets), while source distance achieved the lowest mean AURC (0.1121). The deferred queues were defect-enriched: with RF tree disagreement at 80% coverage, they contained 41.8% of all defective modules and had a mean defect enrichment of 2.10× the target base rate. Selective prediction can therefore reduce automated CPDP risk while routing defect-rich modules to manual review, but it does not remove the need for inspection, and no reliability signal was best on every target.

Downloads

Published

2025-06-30

How to Cite

Muhammad Arif Shah. (2025). SELECTIVE PREDICTION UNDER CROSS-PROJECT SHIFT: ABSTENTION RISK AND DEFECT-ENRICHED REVIEW QUEUES. Spectrum of Engineering Sciences, 3(6), 1304–1329. Retrieved from https://thesesjournal.com/index.php/1/article/view/3975