AI security covers the protection of machine learning models, the data that shapes them and the applications and agents built on top of them.
Much of it reuses familiar controls, but a few properties of large language models break assumptions that traditional security relies on.
This article explains where the two disciplines overlap, where they diverge, which frameworks map the new risks, and how to apply concrete controls and detections to LLM applications and AI agents.
The assumptions that AI breaks
Traditional application security rests on a clear boundary between code and data. SQL injection was solved, in principle, by parameterized queries that keep instructions and user input in separate channels.
An LLM has no such separation: the system prompt, the user’s question and a retrieved web page all arrive as one stream of tokens, and the model decides what to treat as an instruction.
The second broken assumption is determinism. A firewall rule or an access check gives the same answer every time.
A model’s output varies with sampling, context and phrasing, so a guardrail that blocks a malicious request once may not block a reworded version.
Security testing therefore shifts from proving a control works to measuring how often it fails.
The third is the nature of the asset. In a conventional system, the valuable things are data stores and credentials.
In an AI system, the model weights, the training and fine-tuning data, the retrieval index and the agent’s tool permissions are all assets, and each can be stolen, poisoned or abused.
AI security vs traditional cybersecurity, side by side
The table summarizes where established practice carries over and where it needs new methods. Vendors and researchers describe AI security in two senses, using AI to defend systems and defending AI systems themselves; this article focuses on the second.
| Area | Traditional security | AI and LLM security |
|---|---|---|
| Primary injection risk | SQL, command and template injection, fixed with input separation | Prompt injection, with no complete technical fix; mitigated through privilege limits and output controls |
| Supply chain | Libraries, packages, container images | All of those plus pretrained models, datasets, model serialization formats and plugins |
| Integrity attacks | Tampering with code or configuration | Data and model poisoning, backdoored weights |
| Testing | Deterministic tests, scanners, penetration testing | Probabilistic red teaming and evaluation suites, repeated at every model or prompt change |
| Authorization | Users and services with defined roles | Agents acting on behalf of users, often with broad tool access and no human in the loop |
| Threat knowledge base | MITRE ATT&CK, CVE, CWE | MITRE ATLAS, OWASP Top 10 for LLM Applications, AI vulnerability disclosures |
The new attack surface in detail
Prompt injection, direct and indirect
Prompt injection is ranked LLM01 in the OWASP Top 10 for LLM Applications 2025 and tracked in MITRE ATLAS as AML.T0051, with direct, indirect and triggered sub-techniques.
Direct injection is a user typing instructions that override the system prompt. Indirect injection is more dangerous: instructions hidden in an email, document, web page or code comment that the model reads while doing a task.
Indirect injection turns any data source into an attack vector. GBHackers reported on OpenClaw AI agents vulnerable to indirect prompt injection causing data leaks, and on a prompt injection attack that turned a chatbot into a command-and-control agent.
The common thread is an agent that could read untrusted content and act with real permissions.
Model and data supply chain
Public model hubs are the new package registries, with the same risks. Many models are distributed in Python pickle format, which can execute code when loaded.
GBHackers covered malicious ML models found on Hugging Face and the Sleepy Pickle technique, which tampers with a model at load time. OWASP lists this as LLM03 Supply Chain, and ATLAS tracks it as AML.T0010.
Data and model poisoning
Poisoning (OWASP LLM04, ATLAS AML.T0020 Poison Training Data) inserts crafted samples into training, fine-tuning or retrieval data so the model learns a hidden behavior.
Retrieval-augmented generation widens the target: anyone who can write to a wiki page or shared drive that feeds the vector index can influence answers without touching the model.
Excessive agency
OWASP LLM06 Excessive Agency describes agents with more functions, permissions or autonomy than their task needs. This is where AI risk turns into traditional impact.
A report on destructive commands injected into Amazon’s AI coding agent shows how an agent with file system and cloud access can be steered toward deletion.
Leakage and cost abuse
Sensitive information disclosure (LLM02) and system prompt leakage (LLM07) expose secrets or business logic placed in prompts. Unbounded consumption (LLM10) covers denial-of-wallet attacks, where stolen API keys or abusive users run up inference costs.
What stays the same
Most AI incidents still start and end with ordinary weaknesses. Leaked API keys, over-permissioned service accounts, unpatched inference servers and exposed notebooks are classic problems.
CIS Controls v8 asset inventory (Control 1), software inventory (Control 2) and access control management (Control 6) apply directly to models, agents and AI endpoints.
Attackers also use AI against conventional targets. Phishing emails and messages are easier to write convincingly at scale with generative models, but the defenses remain phishing-resistant MFA, user reporting and mail filtering.
Treat “AI-powered attacks” and “attacks on AI” as separate problems with separate owners.
The financial stakes are familiar too. Cybersecurity Ventures projected cybercrime would cost the world $10.5 trillion annually by 2025, and AI systems are now part of that exposure.
IBM’s Cost of a Data Breach Report 2025 found that 13% of organizations reported breaches of AI models or applications, and 97% of those lacked proper AI access controls.
The same report found one in five organizations had a breach linked to shadow AI.
Frameworks that map AI security risk
Three frameworks cover most needs, and they complement rather than compete with each other.
- OWASP Top 10 for LLM Applications 2025 is the developer-facing checklist. Use it for design reviews and secure coding standards for LLM features.
- MITRE ATLAS is the adversary-facing knowledge base, structured like ATT&CK with tactics, techniques and case studies. Use it for threat modeling and to tag detections, just as you tag endpoint rules with ATT&CK IDs.
- NIST AI RMF 1.0, released in January 2023, organizes governance into Govern, Map, Measure and Manage. Its Generative AI Profile, NIST AI 600-1 (2024), lists risks specific to generative systems and suggested actions. Map its outcomes to the Govern function of NIST CSF 2.0 so AI risk sits inside your existing program.
Practitioner example: an indirect injection attack chain
The following scenario is illustrative and combines patterns from public research rather than describing a specific incident.
- A company deploys an email assistant agent that can read the user’s inbox, search the internal wiki and send email on the user’s behalf.
- An attacker sends an ordinary-looking email containing hidden text: instructions telling the assistant to search the wiki for “VPN credentials” and forward the results to an external address (ATLAS AML.T0051.001, indirect prompt injection).
- The user asks the assistant to summarize today’s mail. The model reads the attacker’s email as part of its context and follows the embedded instructions.
- The agent calls the wiki search tool, then the send-mail tool. Both calls are authorized because the agent holds the user’s full delegated permissions (OWASP LLM06).
- Data leaves through a legitimate channel, so data loss prevention keyed to unusual destinations or volumes is the only traditional control likely to notice.
Breaking this chain does not require solving prompt injection. It requires that the agent cannot send email to external domains without user confirmation, and that tools reading untrusted content are separated from tools that take action.
Practitioner example: controls and detections for LLM applications
Constrain agent permissions
Define tool access per agent as a policy enforced outside the model. The snippet below is an illustrative policy format, not a specific product’s syntax.
agent: email-assistant
tools:
read_inbox: { allow: true }
search_wiki: { allow: true, max_results: 5 }
send_email:
allow: true
recipients: internal_domain_only
require_user_confirmation: true
delete_email: { allow: false }
context_rules:
untrusted_sources: [inbound_email, web_fetch]
block_action_tools_after_untrusted_read: true
logging:
log_prompts: true
log_tool_calls: true
retention_days: 90
Secure the model supply chain
- Prefer safetensors or other non-executable formats; refuse pickle files from untrusted sources or load them only in an isolated sandbox.
- Pin models by cryptographic hash or commit, not by name, so a renamed or replaced repository cannot swap the artifact.
- Scan model files with a model-aware scanner and record them in an AI bill of materials alongside the software SBOM.
- Keep fine-tuning and retrieval data under the same write-access controls and change review as source code.
Detect abuse in LLM gateway logs
Route all model calls through a gateway that logs prompts, tool calls and token counts to your SIEM. The illustrative SQL below flags sessions where an agent read untrusted content and then called an outbound action tool within five minutes.
SELECT a.session_id, a.agent_id, a.tool_name AS read_tool,
b.tool_name AS action_tool, b.arguments, b.event_time
FROM llm_tool_calls a
JOIN llm_tool_calls b
ON a.session_id = b.session_id
AND b.event_time BETWEEN a.event_time AND a.event_time + INTERVAL ‘5’ MINUTE
WHERE a.tool_name IN (‘read_inbox’, ‘web_fetch’, ‘read_document’)
AND b.tool_name IN (‘send_email’, ‘http_post’, ‘write_file’, ‘run_shell’)
ORDER BY b.event_time DESC;
Add alerts for token usage spikes per API key (unbounded consumption), for responses containing strings from the system prompt (prompt leakage), and for model invocations from new IP ranges, which often indicate stolen keys.
Test continuously
Red team every significant change to the model, system prompt, tools or retrieval sources, because any of these can reopen a closed weakness.
Keep a regression suite of known injection payloads and measure the failure rate over time rather than recording a single pass or fail.
Building an AI security program on existing foundations
Start with inventory: which models, AI services, agents and API keys exist, who owns them and what data they touch. IBM’s shadow AI findings suggest many organizations cannot answer that today.
Then assign ownership, because AI features often sit between data science, product and security teams with nobody accountable.
Extend existing processes rather than creating parallel ones. Add OWASP LLM risks to threat modeling templates, ATLAS techniques to the detection backlog, model artifacts to vulnerability management and AI agents to access reviews.
The practices are familiar; the new work is applying them to assets most security programs were not built to see.
Key takeaways
- AI security differs from traditional security because LLMs mix instructions and data, behave probabilistically and introduce new assets such as weights, training data and agent permissions.
- Prompt injection has no complete fix, so limit what a manipulated model can do through least-privilege tools and human confirmation for high-impact actions.
- Treat model hubs like package registries: pin by hash, avoid executable formats and scan artifacts.
- Use OWASP Top 10 for LLM Applications for design, MITRE ATLAS for threat modeling and detection, and NIST AI RMF for governance.
- Log prompts, tool calls and token usage through a gateway so the SOC can detect AI abuse with the same workflows it already runs.
FAQ
How is AI security different from traditional cybersecurity?
Traditional security protects deterministic systems where code and data are separate. AI security must also handle models that treat any input as potential instructions, probabilistic behavior, poisoned training data, malicious model files and autonomous agents with delegated permissions.
What is the OWASP Top 10 for LLM applications?
It is OWASP’s list of the most critical risks for applications built on large language models.
The 2025 edition covers prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption.
What is MITRE ATLAS used for?
MITRE ATLAS is a knowledge base of adversary tactics and techniques against AI systems, modeled on ATT&CK. Security teams use it for threat modeling, red team planning and tagging detections for attacks such as prompt injection and training data poisoning.
Can prompt injection be fully prevented?
No reliable method fully prevents it today. Input filters and model hardening reduce the success rate, but the main defense is limiting impact: least-privilege tools, separation of untrusted content from action tools and human approval for sensitive steps.
How do you secure AI agents in the enterprise?
Give each agent its own identity with minimal permissions, enforce tool policies outside the model, require confirmation for external or destructive actions, log every tool call to the SIEM and include agents in regular access reviews.
