WHEN COVERAGE IS NOT ENOUGH: THE RELIABILITY–INFORMATIVENESS TRADE-OFF IN CROSS-PROJECT CONFORMAL BUILD-FAILURE PREDICTION

Abstract

Conformal prediction turns a classifier’s scores into prediction sets with a nominal coverage guarantee, which makes it attractive for continuous integration (CI), where a build-failure predictor trained on other projects may have to be deployed on a project with little history. The guarantee assumes exchangeability between calibration and deployment data, which cross-project deployment violates. This paper reports a staged audit of label-conditional split conformal build-failure prediction on TravisTorrent (619,395 resolved builds from 1,283 projects) with logistic regression (LR), random forest (RF), and histogram gradient boosting (HGB) on 23 pre-build features. A six-project discovery stage tested the frozen hypothesis that cross-project deployment causes systematic marginal undercoverage at α = 0.10. It failed for all three classifiers: median coverage fell from 0.875 to 0.783 for LR but rose from 0.916 to 1.000 for RF and from 0.891 to 0.984 for HGB. Because coverage appeared to be preserved through two-label sets, the endpoint was redesigned as the correct singleton rate (CSR) and tested on 30 untouched projects under a frozen rule. The redesigned hypothesis did not replicate. Median CSR fell from 0.162 to 0.049 (LR), 0.103 to 0.009 (RF), and 0.111 to 0.024 (HGB), but no classifier met all conditions: the LR loss was positive in only 20 of 30 projects (Holm-adjusted p = 0.099), the RF loss (0.075; 24 of 30; Holm p = 0.005) fell below the 0.10 effect criterion, and the HGB loss (0.062; Holm p = 0.221) failed all conditions. Descriptively, median coverage rose under cross-project deployment for every classifier while the median two-label rate rose to 0.908, 0.981, and 0.917; in 7 (LR), 1 (RF), and 3 (HGB) targets, however, coverage fell below 0.50. In a post-confirmatory robustness stage, coverage gain and ambiguity gain were strongly correlated across targets (Spearman ρ = 0.70 to 0.82); the pattern persisted across significance levels and calibration sizes for RF but not for LR and HGB, and static feature-mean shift was not a consistent correlate. Within-project ranking was weak (median PR-AUC 0.22 to 0.26), which bounds all conclusions. For these features, nominal coverage alone overstated the usefulness of cross-project conformal build-failure prediction, but the predicted loss of informativeness did not replicate under its frozen criterion.