STABLE DISCRIMINATION, SHIFTING OPERATING POINTS: TEMPORAL TRANSPORTABILITY OF NON-LABORATORY DIABETES SCREENING

Abstract

Diabetes screening models act through a fixed threshold, yet they are usually validated by discrimination alone. We examined whether source-selected operating points of non-laboratory diabetes screening models transport across time when discrimination is preserved. Using public-use NHANES data, we studied adults aged ≥20 years without a reported prior diabetes diagnosis; HbA1c ≥6.5% defined laboratory-defined probable undiagnosed diabetes. Logistic regression, random forest and histogram gradient boosting used six non-laboratory predictors (age, sex, body mass index, waist circumference, self-reported hypertension, smoking status). Models were developed on 2009–2014, and thresholds were selected from leave-one-cycle-out predictions at survey-weighted sensitivity targets of 0.80–0.95. A pattern found in 2015–2016 was then tested in August 2021–August 2023 (4,698 adults; 116 HbA1c-positive) against a rule frozen beforehand: at the 0.90 threshold, absolute AUROC change ≤0.05, sensitivity ≥0.90, a higher proportion classified positive and lower specificity, with at least two of three model families required to pass. The prespecified criterion was met by logistic regression and random forest. Their AUROC changed from 0.780 to 0.793 and from 0.756 to 0.793 and sensitivity remained 0.949 and 0.971, while the proportion classified positive rose by 3.7 and 1.6 percentage points. Histogram gradient boosting showed the largest operating-point shift (+12.9 points) but did not meet the criterion because AUROC rose by 0.066. The same direction appeared at all 12 model–target combinations. In post-confirmatory analyses, intercept recalibration on 2017–2018 data restored some selectivity for the tree ensembles but reduced logistic-regression sensitivity to 0.807. Source-selected screening operating points did not fully transport across time even when discrimination and sensitivity remained acceptable; temporal validation should report calibration and screening volume alongside AUROC.