54% of AI-generated patches don't work. Here's what you're missing.
The research
1Password presented findings at Black Hat USA 2026 covering the evaluation of more than 6,000 AI-generated patches for documented security vulnerabilities. The result: 54% of those patches failed to fix the original vulnerability. Beyond that failure rate, a portion of the evaluated patches introduced new security flaws while attempting to address the original issue.
The research does not single out a specific model or tool. The finding applies broadly across AI-assisted code generation applied to security remediation tasks.
Why this number is significant
Security teams and DevSecOps platforms have been adopting AI-assisted patching at pace. The workflow is intuitive: a scanner finds a vulnerability, an AI tool generates a fix, a developer reviews and merges. The appeal is speed. The assumption embedded in that workflow is that AI-generated patches are meaningfully correct most of the time.
54% failure rate means that assumption is wrong. If half of your AI-generated patches do not fix the underlying issue, you have a false closure problem. Tickets close. SLA metrics improve. Dashboards turn green. The vulnerability remains.
What causes AI patches to fail
The first pattern is surface-level matching. AI models trained on code patterns will often generate a patch that addresses the syntactic signature of a vulnerability class without understanding the semantic root cause. A SQL injection fix that escapes a single query parameter while leaving a second unescaped input path unchanged will pass a code review of the modified lines and fail on a regression test of the full attack surface.
The second is context blindness. A patch that is correct for the isolated function under review may be incorrect in the context of the broader codebase. The AI has a limited view. The vulnerability may live in the interaction between components not visible in the local context window.
The third is the introduced-flaw problem. Patches that rewrite logic to remove a vulnerability may introduce memory handling errors, race conditions, or input validation gaps. The original vulnerability is removed. A new one is present.
What a reliable AI-assisted patching workflow looks like
Treat AI-generated patches as drafts, not solutions. Require test coverage: the patch should be accompanied by tests that reproduce the original vulnerability and confirm it no longer triggers. Add AI-assisted review to AI-assisted generation. Use a second AI pass specifically to evaluate the patch for correctness and new vulnerability introduction. Track your false-closure rate. If you have AI-assisted patching in production, measure how often patched vulnerabilities resurface or remain exploitable post-patch.
The bottom line
AI-assisted patching saves time when it works and creates risk when it does not. The 1Password research establishes that it fails to work the majority of the time without validation. The right response is not to stop using AI in remediation workflows. The right response is to add the validation steps that most current implementations skip.
Gigia Tsiklauri is a Security Architect and founder of Infosec.ge. Get in touch if you are building or evaluating AI-assisted security tooling and want a practitioner perspective.