WHEN THRESHOLDS DO NOT TRANSFER: DECISION-THRESHOLD PORTABILITY IN CROSS-PROJECT DEFECT PREDICTION

Authors

  • Muhammad Arif Shah

Keywords:

Cross-project defect prediction; Decision threshold; Threshold portability; Matthews correlation coefficient; Distribution shift; Class imbalance; Empirical software engineering; Software quality assurance.

Abstract

Cross-project defect prediction (CPDP) models output defect probabilities, but deployment requires a decision threshold, which is usually left at 0.50 and rarely examined. Whether a threshold that works on source projects transfers to an unseen target project has received comparatively limited empirical attention. We evaluate the fixed 0.50 threshold and five source-only threshold policies (pooled MCC, pooled balanced accuracy, pooled F1, median-project MCC and minimax-project MCC) with Logistic Regression, Random Forest (RF) and HistGradientBoosting under leave-one-project-out validation. Thresholds are selected through nested leave-one-source-project-out validation without target labels, after exact deduplication and removal of exact source–target feature-vector overlap. A retrospective target-optimal threshold is used only to quantify MCC regret. In a three-stage study, source-only policies reduced regret and raised defect recall on AEEEM and NASA PROMISE and on an expanded 18-target benchmark (RF mean MCC 0.1848 at 0.50 versus 0.2618 for pooled MCC; mean regret 0.1047 versus 0.0276). In a curation control with eight canonical Jureczko/PROMISE projects, however, the same policies often raised recall while lowering MCC. For RF and Logistic Regression, the fixed 0.50 threshold had the highest mean MCC (RF 0.1759 versus 0.1128 for pooled MCC), although robust policies outperformed it on several individual targets. Neither the 0.50 default nor any single source-only policy was reliable across benchmark families and models. Therefore, treat threshold selection as an explicit cross-project deployment decision and report it transparently in CPDP studies.

Downloads

Published

2025-08-26

How to Cite

Muhammad Arif Shah. (2025). WHEN THRESHOLDS DO NOT TRANSFER: DECISION-THRESHOLD PORTABILITY IN CROSS-PROJECT DEFECT PREDICTION. Spectrum of Engineering Sciences, 3(8), 1523–1547. Retrieved from https://thesesjournal.com/index.php/1/article/view/3977