AI-ASSISTED AUTOMATED SOFTWARE TESTING AND VULNERABILITY DETECTION USING LARGE LANGUAGE MODELS

Abstract

The integration of Large Language Models (LLMs) into software engineering has fundamentally disrupted traditional paradigms of software quality assurance and cybersecurity. This review systematically synthesizes recent empirical advances in LLM-assisted automated software testing and vulnerability detection, evaluating both the transformative capabilities and critical limitations of these emerging technologies. The analysis examines four principal domains: LLM-powered test case generation and fuzzing, static vulnerability discovery, automated vulnerability repair (AVR), and agentic patching frameworks. Empirical evidence demonstrates that LLM-based test generators, including ChatUniTest, CoverUp, and TestPilot, achieve superior branch coverage and mutation scores compared to search-based tools like EvoSuite, with frameworks such as HITS improving coverage from 68.0% to 82.3%. In vulnerability detection, hybrid neuro-symbolic workflows combining traditional SAST tools with LLM verification filters reduce false-positive rates by up to 93.3%, while specialized models like VulnScout-C demonstrate balanced precision-recall trade-offs on dual-verified datasets. Autonomous agentic frameworks, including SWE-agent and OpenHands, achieve 72-74% issue resolution rates on SWE-bench Verified, with the SAN2PATCH framework attaining 79.5% patch success rates. However, this review identifies a critical security correctness disconnect: despite high functional pass rates, AI agents achieve only 10.5-12.5% security correctness on rigorous evaluations, frequently reintroducing insecure legacy functions to satisfy functional tests. Furthermore, agentic workflows exhibit susceptibility to indirect prompt injection and flow-control vulnerabilities, with attack success rates reaching 40.7%. Data contamination in public benchmarks artificially inflates performance metrics, compromising evaluation integrity. The review concludes that while LLMs offer unprecedented semantic reasoning capabilities, their probabilistic nature necessitates robust verification pipelines, hybrid architectures, and security-aware training paradigms to ensure trustworthy deployment across the software development life cycle.