THRESHOLD-DEPENDENT BACKWARD COMPATIBILITY UNDER TEMPORAL RETRAINING IN JUST-IN-TIME SOFTWARE DEFECT PREDICTION

Abstract

Just-in-time software defect prediction (JIT-SDP) models are periodically retrained as new commit history accumulates. Retraining is usually judged by aggregate metrics such as ROC-AUC or average precision. An updated model can, however, improve on aggregate while becoming wrong on commits that the previous model classified correctly. This study characterises backward compatibility between successive temporally retrained JIT-SDP models, examines how model-update regressions depend on the operating threshold and the error class, and evaluates whether simple compatibility-aware update policies can reduce negative flips without suppressing too many beneficial prediction changes. Using six open-source projects from a public commit-level JIT-SDP dataset (118,877 commits), we compared OLD and NEW models trained on expanding chronological histories and scored on the identical next window, for Logistic Regression, Random Forest, and Histogram Gradient Boosting at five thresholds (0.30 to 0.70). The study proceeded in three stages: discovery and redesign on three projects, a pre-specified independent replication on three further projects, and an analysis of six fixed compatibility-aware policies on all six projects against a pre-specified four-part success rule. Negative flips were pervasive. Among comparisons in which the updated model did not degrade ROC-AUC by more than 0.01, 99.7% contained at least one negative flip and 21.5% had a negative flip rate (NFR) of at least 5% at the central thresholds. The replication confirmed strong threshold dependence: mean NFR fell from 0.0342 at threshold 0.30 to 0.0232 at 0.70, while the composition of regressions shifted from newly introduced false alarms towards newly missed defects. Regression levels differed markedly by project and learner. Probability interpolation and near-threshold fallback reduced NFR by 18.1% to 82.2%, but stronger protection also suppressed positive flips; no non-trivial policy met all four pre-specified criteria (verdict: mitigation not supported). Aggregate improvement did not ensure preservation of previously correct commit-level decisions. Backward compatibility in JIT-SDP behaved as an operating-point property, and the tested policies exposed a compatibility-adaptation trade-off rather than a cost-free remedy.