Sep 12, 2023 Trustworthy AI

56% More AI Security Incidents - And We're Still Calling This 'Early Days'

AI security incidents rose 56% in a year, and the industry response was to call it growing pains. At some point, the "early days" framing becomes a way to avoid accountability. The incidents aren't anomalies. They're the system working as designed.

By September 2023, the narrative that enterprise leaders told themselves about AI security went something like this: AI is powerful but new. We’re in the early innings. The risks are real but manageable. Give us time: we’re working on it.

The data told a different story. The Stanford HAI AI Index Report, tracking incidents logged to the AI Incident Database, documented 123 AI-related security incidents in 2023; a 32.3% increase over 2022. The following year, that number jumped to 233 incidents, a 56.4% year-over-year acceleration. The AI Incident Database, which had been tracking these events since 2013, showed a growth curve that had gone over twentyfold in a decade.

This was not “early days.” This was an exponential incident curve meeting an industry that hadn’t even created the tracking category yet. Most enterprise security teams in September 2023 had no “AI incident” field in their SIEM. No AI-specific playbook in their incident response program. No AI security line item in their annual budget. They were watching the numbers climb while organizing their defenses around a threat model that didn’t include the fastest-growing attack surface in their environment.

The measurement problem nobody fixed

The first problem was definitional. What counted as an “AI security incident”?

Traditional security incident taxonomies, unauthorized access, data breach, malware infection, denial of service, were designed for systems with deterministic behavior. You could draw a line between intended behavior and anomalous behavior because the system had specifications. An AI system that generates a discriminatory hiring recommendation, leaks training data in a chat response, or gets manipulated through prompt injection does not fit cleanly into any of these categories.

The AI Incident Database, maintained by the Responsible AI Collaborative, attempted to bridge this gap. Its taxonomy included categories that traditional security frameworks ignored entirely: algorithmic discrimination, privacy violations through model behavior, safety failures in autonomous systems, and deepfake-enabled fraud. The incidents it captured ranged from AI-generated sexually explicit deepfakes of real people to automated hiring systems that systematically discriminated against protected classes to chatbots that encouraged self-harm in conversations with vulnerable users.

Each of these was a security incident in any meaningful sense. Most would not have been captured by a traditional enterprise SIEM, because the SIEM was looking for network anomalies and malware signatures, not for an AI system providing dangerous medical advice or leaking PII through a creative writing exercise.

McKinsey’s Global AI Survey data reflected this measurement gap from the enterprise side. Organizations consistently identified their top AI risk concerns: inaccuracy (64% of respondents), regulatory compliance (63%), and cybersecurity (60%). But only roughly 20% of companies had actual risk policies in place for generative AI. The gap between recognizing the risk and doing anything about it was not a planning failure. It was a category failure. Teams didn’t know where to put AI security in their existing frameworks, so they didn’t put it anywhere.

The deepfake acceleration

If one category of AI security incident defined 2023, it was deepfakes. Not because the technology was new, face-swapping tools had existed for years, but because the quality and accessibility crossed a threshold that made deepfakes operationally useful for criminals at scale.

Identity verification firms tracked the acceleration in real time. Sumsub reported a 3,000% surge in deepfake-related fraud attempts during 2023. Onfido’s data corroborated the trend. The attacks were no longer limited to crude face-swaps on video calls. They included synthetic voice cloning used for CEO fraud, AI-generated documents for identity verification bypass, and deepfake video used to impersonate executives in wire transfer authorizations.

Matthew Walsh, a researcher involved in the OWASP AI security taxonomy, noted that “businesses across various sectors, government agencies, and private citizens have become targets of these attacks”, a statement that captured the breadth of the surface area but understated the speed at which it was expanding.

The regulatory response lagged. By 2024, 24 U.S. states had passed deepfake-related legislation, but the laws were narrowly targeted; focused primarily on election interference and non-consensual intimate imagery rather than on the business fraud and identity compromise patterns that were actually driving financial losses. The regulatory apparatus was fighting the last war while the current one accelerated.

For enterprise security teams, deepfakes represented something qualitatively different from traditional social engineering. A phishing email can be analyzed: headers, sender reputation, URL analysis, content patterns. A deepfake video call has no equivalent forensic artifact. The “evidence” is the victim’s memory of a conversation that appeared real. There is no malware signature, no network log, no digital fingerprint. The attack surface is human perception itself, and the defense mechanisms that enterprises had built, email gateways, URL filters, endpoint detection, were architecturally irrelevant.

This gap between the threat vector and the defense architecture became a defining challenge. Security teams that had spent decades building detection capabilities for digital artifacts suddenly faced attacks that exploited analog trust: the trust humans place in faces, voices, and real-time interaction. The OWASP LLM Top 10, published the same month, didn’t include deepfakes directly because they fell outside the LLM application scope. But the incident data showed that generative AI’s most immediate financial impact on enterprise security wasn’t prompt injection or data poisoning. It was fraud enabled by AI-generated media that fooled humans at the interface layer.

Trust erosion as a compound risk

The security incidents themselves caused direct harm. But the compound effect, the erosion of trust in AI systems, created a second-order risk that was harder to measure and harder to reverse.

Stanford HAI’s data showed public trust in AI companies to protect personal data fell from 50% in 2023 to 47% in 2024. A three-percentage-point drop sounds modest until you consider that it represents a directional shift during a period when AI companies were investing billions in trust-building campaigns.

For enterprises, trust erosion translated directly into operational friction. Customers became more reluctant to interact with AI-powered support systems. Employees questioned whether AI-generated analysis could be relied upon. Boards demanded more reporting on AI risk; not because they understood the technical specifics, but because the headlines made the risk feel real.

The McKinsey data captured this tension perfectly: organizations were simultaneously increasing AI investment (92% planned to invest more) and failing to scale AI beyond pilots (only 1% described their AI maturity as advanced). The investment was flowing toward a technology that the organization didn’t fully trust. This wasn’t irrational; it was an accurate reflection of a market where the potential was enormous and the risks were unquantified.

The budget allocation failure

By the time the 2023 incident data was being compiled, the misalignment between AI security spending and AI security risk had become structural.

Enterprise security budgets in 2023 were still organized around traditional categories: network security, endpoint protection, identity and access management, cloud security, application security. AI security, when it appeared at all, was a footnote under “emerging threats” or buried within application security as a subcategory.

But the incident data showed that AI-specific threats were growing faster than any traditional category. The 32.3% year-over-year increase in 2023 and the 56.4% increase that followed in 2024 outpaced the growth rate of ransomware, phishing, and supply chain attacks: the categories that consumed the majority of security budgets.

The U.S. federal government’s regulatory response tracked the same acceleration. Federal agencies issued 59 AI-related regulations in 2024, more than double the 25 issued in 2023. Legislative mentions of AI increased 21.3% across 75 countries globally. The regulatory landscape was shifting from “monitor” to “mandate,” and enterprises without dedicated AI security programs were about to discover that retrofitting is always more expensive than building.

What I see from inside enterprise AI deployments

Architecting AI systems for 170,000 users has given me a front-row seat to the gap between AI incident data and enterprise preparedness. Three patterns stand out.

The first is the incident detection gap. Traditional security monitoring tools are blind to AI-specific threats. A prompt injection attack that causes an LLM to exfiltrate data through its generated output doesn’t trigger a network IDS rule. A training data extraction attack doesn’t set off a DLP alert because the data is being generated by the model, not copied from a database. An AI system that produces discriminatory outcomes doesn’t appear in any security dashboard because discrimination isn’t a log event.

The second is the response gap. Even when AI incidents are detected, incident response teams don’t have playbooks for them. How do you contain a prompt injection when the model is a shared service? How do you perform forensics on an AI system that doesn’t produce traditional logs? How do you communicate to executives that the breach vector was “someone asked the chatbot the right question”?

The third is the attribution gap. Traditional incidents have clear attack chains: initial access, lateral movement, data exfiltration. AI incidents often lack this structure. An employee leaking source code through ChatGPT is not an attacker, they’re a user. A deepfake-enabled wire fraud doesn’t leave malware signatures. An AI system that produces biased outcomes isn’t compromised, it’s functioning as designed, just with training data that encoded societal biases. The concepts of “attacker” and “vulnerability” don’t map cleanly to AI systems, and security teams organized around those concepts struggle to respond.

Through my work on the ACM AISec Program Committee, I’ve reviewed dozens of research papers attempting to formalize AI threat taxonomies. The academic community is making progress on the categorization problem. But the gap between academic taxonomy and enterprise SIEM configuration remains vast.

The fourth pattern, the one I find most concerning, is the normalization gap. As AI incidents accumulate, organizations develop a tolerance for them. A chatbot hallucinating a product feature that doesn’t exist becomes “something we’re working on” rather than an incident. An AI system that occasionally surfaces inappropriate content in customer interactions becomes “an edge case” rather than a data quality failure. An employee using unsanctioned AI tools to process customer data becomes “shadow IT” rather than a data breach.

This normalization is dangerous because it suppresses the signal that should trigger investment. If your AI systems are producing outputs that would constitute incidents under any reasonable taxonomy, but you’re not counting them as incidents because you don’t have the category, your risk reporting is telling leadership that AI is less risky than it actually is. Investment follows reporting. Underreporting drives underinvestment. Underinvestment drives more incidents. The cycle is self-reinforcing.

What to do about it Monday morning

The incident curve is not going to flatten on its own. Budget, staffing, and process decisions need to reflect the threat data that already exists.

Create an “AI incident” category in your security incident management system. Today. If your SIEM, SOAR platform, and incident tracking tools don’t have a way to tag and track AI-specific incidents, you are guaranteed to undercount them. And what you undercount, you underfund. Define the subcategories: prompt injection, data leakage through AI, deepfake-enabled fraud, AI system misuse, model supply chain compromise. Start counting.

Review your cyber insurance policy for AI-specific exclusions. Most commercial cyber insurance policies written before 2024 contain either explicit exclusions for AI-related incidents or ambiguous language that insurers will use to deny AI-related claims. If your policy doesn’t explicitly cover AI-generated fraud, deepfake-enabled social engineering, and data leakage through LLM interactions, you have an uninsured exposure.

Add AI security metrics to your board risk reporting. Boards are already asking about AI risk. Give them quantitative answers: number of AI-specific incidents detected, percentage of LLM interactions that trigger DLP rules, count of unsanctioned AI tools detected in network traffic. If you can’t produce these numbers, that itself is the board-level finding.

Establish an AI incident response playbook separate from your traditional IR program. The playbook should address containment for shared AI services, forensics for non-deterministic systems, and communication templates for AI-specific breach scenarios. Staff it with people who understand both security operations and how LLMs actually work.

Benchmark your AI incident rate against the Stanford HAI data. If your organization reports zero AI incidents while the industry average is climbing 50% per year, you don’t have an AI-secure environment. You have an AI-blind one. The incidents are happening. The only question is whether you’re counting them.