WHEN FLAKY-TEST PREDICTORS FAIL TO TRANSFER: RANKING COLLAPSE UNDER CROSS-PROJECT DEPLOYMENT

Abstract

Flaky-test predictors built on static test-code metrics can score highly when they are trained and evaluated within a single project, yet the situation that motivates cross-project prediction, a project with too little labelled history of its own, requires a model to be deployed on a project it has never seen. This study audits that deployment step on the public 29-project cross-project flaky-test dataset (87,875 tests, 1,385 of them flaky), audited at a fixed repository commit, using standardized logistic regression (LR), random forest (RF), and histogram gradient boosting (HGB) classifiers with 25 static predictors. A five-project discovery stage tested a pre-specified feature-family fragility hypothesis. That hypothesis failed (1 of 5 projects passed; 3 were required), but the discovery data showed a large collapse from within-project to leave-one-project-out (LOPO) performance. The study was therefore redesigned, and the redesigned hypotheses were tested on the 24 remaining projects, with all discovery projects excluded from both training and targets and with decision rules frozen before any replication outcome was viewed. Cross-project transfer failure replicated: the median within-project minus LOPO flaky-class F1 was 0.800 for RF and 0.720 for HGB, meeting the frozen criterion, whereas the LR drop of 0.093 was statistically significant but fell below the frozen 0.10 effect threshold. The loss was dominated by ranking collapse. Median PR-AUC fell from 0.343 to 0.089 (LR), from 0.935 to 0.025 (RF), and from 0.903 to 0.035 (HGB), with losses in 23 or 24 of 24 projects (Holm-adjusted p ≤ 2.5 × 10⁻⁶). Decision thresholds selected exclusively on source projects removed the zero-positive output that RF produced at threshold 0.50 in 23 of 24 targets, but median F1 gains were only −0.004 (LR), 0.020 (RF), and 0.001 (HGB), and none met the frozen 0.05 criterion. A post-confirmatory robustness stage found the ranking collapse for all three models under repeated within-project baselines, among targets with at least 30 flaky tests, in both low- and high-prevalence strata, and under Recall@K and Recall@1% review budgets. Alternative source-only threshold objectives did not restore performance, and static feature-mean shift was not associated with the size of the loss. For these predictors and this benchmark, strong within-project scores did not indicate usable cross-project rankings, and threshold recalibration alone could not repair a ranking that had not transferred.