SOURCE-POOL FRAGILITY IN CROSS-PROJECT DEFECT PREDICTION: PREDICTION INSTABILITY, SOURCE INFLUENCE, AND TARGET-BLIND HARM DETECTION

Abstract

Multi-source cross-project defect prediction (CPDP) trains a model on several historical projects and applies it to an unseen target project. Which projects make up that source pool is a training-data provenance decision, but it is rarely examined as a source of instability in the predictions practitioners actually consume. We quantify how strongly target-project predictions depend on the exact composition of the source pool, whether some historical projects exert disproportionate influence, and whether harmful pool changes can be flagged without target labels. We conducted a three-stage empirical study over three public benchmark families (AEEEM, NASA PROMISE, and Jureczko/PROMISE; 18 target projects) and three learners (Logistic Regression, Random Forest, and HistGradientBoosting). The primary manipulation is DROP-ONE: removing a single source project from the all-source pool and retraining (288 removals in total). We measured target-label-free changes in predicted probabilities, risk rankings, top-20% review sets, and decisions, and used target labels only retrospectively to measure ROC-AUC and Average Precision changes. We then evaluated pre-fit and post-fit label-free indicators as detectors of removals that reduced ROC-AUC by more than a descriptive 0.03 cutoff, first with leave-one-target-out validation and then with leave-one-benchmark-family-out validation. Source-pool composition materially changed target predictions. In 50 of 54 target-learner settings (92.6%), at least one removal left less than 75% Jaccard overlap in the top-20% review set, and in 24 of 54 settings (44.4%) at least one removal reduced ROC-AUC by more than 0.03, although only 33 of 288 individual removals (11.5%) did so. Review sets often changed substantially even when ROC-AUC barely moved. Influence was concentrated: in NASA PROMISE, JM1 produced the largest probability shift in every setting where it was a source, a pattern that coincides with its dominant share of source rows. Under leave-one-family-out transfer, post-fit instability indicators gave the highest macro-family harm-detection ROC-AUC for all three learners at the 0.03 cutoff (0.666, 0.743, and 0.742 for LR, RF, and HGB). The signal was weak for the largest held-out family, varied with the cutoff, and did not hold under pooled evaluation for RF. Multi-source CPDP can depend substantially on which historical projects are in the training pool. Source-pool provenance should be documented and monitored as part of CPDP trustworthiness.