AUDITING SEHAT: RELIABILITY, SAFETY, AND LANGUAGE-COVERAGE GAPS IN A MULTILINGUAL AI HEALTH TRIAGE AGENT FOR PAKISTAN

Abstract

Large language models are increasingly proposed as front-line triage assistants, yet most published evaluations are conducted in English and rarely examine the non-LLM components that ship alongside the conversational agent, or test the whole pipeline end-to-end against real, external data. This paper reports a full source-level audit of Sehat, a multilingual LLM triage and hospital-recommendation system built for Pakistan and intended to serve Urdu, Roman Urdu, Pashto, and Punjabi speakers. We show that auditing an entire pipeline, rather than its LLM component alone, surfaces reliability and safety problems that otherwise stay hidden. The brain-tumor MRI screening module's classifier is a from-scratch, never-validated pixel-intensity model; a second module built on top of it fabricates a specific tumor size, location, and type from a brightness threshold with no connection to any trained model. The emergency-detection safety filter is a static English/Roman-Urdu keyword list with zero coverage for Urdu script, Pashto, or Punjabi, the very languages, alongside Roman Urdu, the system claims to serve. We replicated the deployed classifier exactly and benchmarked it, under patient-grouped cross-validation and a paired bootstrap ΔAUROC test, against classical machine-learning alternatives (HOG with logistic regression and a random forest) and a fine-tuned, ImageNet-pretrained ResNet-18 obtained partway through the study. Both HOG+Random Forest and ResNet-18 significantly outperform the deployed baseline by ΔAUROC (+0.117 and +0.185 respectively, 95% CI excluding zero), a conclusion McNemar's test alone had understated; patient-grouped cross-validation, however, shows their advantage over each other is not stable (0.719 ± 0.078 vs. 0.683 ± 0.064), and evaluation on a second, independently sourced external dataset (n=253) shows every model's performance collapses, two of the four falling below chance, the most important caution in this paper's vision-pipeline evidence. Investigating a discrepancy between pooled and grouped cross-validation, we independently uncovered and quantified a patient-level data-leakage bug: naive random cross-validation inflates a classical model's AUROC from a corrected 0.719 to an uncorrected 0.867. We additionally audited the emergency-keyword list's language coverage directly from source (18 of 20 keywords English, 2 Roman Urdu, zero in the three other target languages), corroborated the associated recall risk on a small English-only methodology-validation pilot (keyword recall 0.333 vs. 0.833 for a classical text classifier on 12 indirectly phrased emergencies), and recomputed the system's hospital-recommendation policy on both its original 10-hospital registry and an independently compiled, threefold-larger 15-hospital, 9-city registry, finding the government-first policy adds 0.11–0.19 km of mean travel distance across both. For the parts of the audit that require clinician annotation or live LLM access we did not have, a multilingual recall benchmark, clinician-rated dialogue and retrieval-grounding evaluation, we specify a complete, pre-registered protocol and provide working prototype code for the adaptive-stopping and retrieval-grounding components, while explicitly declining to fabricate clinical validity we do not have. We close with a Threats to Validity analysis and the ethical and procedural steps we believe are necessary before a system like this moves from prototype toward real deployment.