CROSS-VENDOR VULNERABILITY SEVERITY PREDICTION: SAFETY, SELECTIVITY, AND THE COST OF RECALL PRESERVATION

Abstract

Text-based vulnerability severity predictors are usually evaluated on pooled data, whereas a deployed triage model must often score vulnerabilities of vendors it has never seen. This study examines one costly error under such deployment, the unsafe downgrade of a truly HIGH or CRITICAL vulnerability to LOW or MEDIUM; in this paper “safety” refers only to severe-class recall and the avoidance of such downgrades. The data are CVEs published on or before 28 February 2026 that affect a single application vendor (147,990 CVEs from 19,620 vendors), represented using metadata from a later NVD JSON 2.0 snapshot. TF-IDF logistic regression (LR), linear support vector machine (SVM), and complement naive Bayes (CNB) classifiers were applied to descriptions with CVSS strings and severity words masked, and the vendor was the unit of inference. A nine-vendor discovery stage tested whether unseen-vendor deployment increases the unsafe downgrade rate (UDR) relative to a within-vendor model; only LR met the frozen rule, and the stage returned STOP_OR_REDESIGN. A redesigned hypothesis, that decision thresholds selected on source-vendor calibration data to achieve severe recall of at least 0.90 would fail to keep target UDR at or below 0.10 while ranking quality remained materially preserved, was tested in an independent untouched-vendor replication on 30 vendors and was not replicated: no model met the frozen rule (Holm-adjusted p = 1.000 for all three), median target UDR was 0.035 (LR), 0.085 (SVM), and 0.000 (CNB), and the 0.10 limit was exceeded in 7, 12, and 2 of 30 vendors. A post-confirmatory, descriptive analysis showed that high severe recall coincided with low selectivity: at the 0.90 recall target, median target specificity was 0.251, 0.274, and 0.148, and the models labelled 86%, 81%, and 92% of target CVEs as severe against a median severe prevalence of 49%. At a 0.95 target, median specificity fell to 0.158, 0.160, and 0.015. Over-triage was associated with targets whose severe prevalence was lower than that of the source calibration pool (Spearman ρ from −0.70 to −0.83; unadjusted). Evaluations of cross-vendor severity triage should therefore report specificity and the predicted-severe rate alongside severe recall, because recall alone can conceal a substantial potential triage burden.