Module 3.3 — OWASP LLM Top 10 (2025) as Defender’s Checklist

50-minute lecture · Day 3 afternoon

Learning objectives

By end of this module, students can:

1. Name and describe all ten entries in the OWASP LLM Top 10 (2025) — the canonical taxonomy of LLM-application risks
2. Map each OWASP LLM entry to a concrete detection-engineering deliverable (Sigma rule, YARA rule, SIEM monitoring pattern, or architectural control)
3. Identify which of the ten risks are most operationally relevant to a SOC defending enterprise copilots vs internal RAG bots vs agentic workflows
4. Use OWASP LLM Top 10 as a self-audit framework against the org’s existing LLM deployments

The taxonomy

OWASP publishes the LLM Top 10 as the de-facto industry taxonomy for LLM application security. The 2025 revision (available at owasp.org/www-project-top-10-for-large-language-model-applications/) is the canonical reference.

The detection engineer’s value-add is to translate each OWASP entry into something the SOC can actually deploy and measure. This module does that translation.

The ten entries with detection-engineering deliverables

LLM01 — Prompt Injection

Description: Malicious inputs (direct or indirect) that manipulate the model’s behavior to bypass guardrails or execute unauthorized actions.

Canonical example: An attacker places a hidden prompt on a webpage which, when summarized by an LLM assistant, tricks the assistant into exfiltrating private data or altering behavior (e.g., Bing Chat indirect prompt injection cases from 2023; EchoLeak CVE-2025-32711 from 2025 — covered in Module 3.4).

Detection deliverable: Sigma rule matching known jailbreak-string patterns (ignore all previous instructions, DAN mode, role-confusion strings) in inbound API request payloads. The Codex-generated prompt_injection_detector.py in your course materials implements this. Pair with output filtering (Module 3.5 guardrails).

LLM02 — Sensitive Information Disclosure

Description: Unintentional exposure of PII, proprietary data, or credentials through model outputs or training data.

Canonical example: The Samsung incident (2023) where engineers pasted proprietary source code into ChatGPT to seek coding help — and the code was exposed via the chatbot’s training pipeline.

Detection deliverable: DLP regex/YARA rules detecting high-entropy secrets, AWS-key patterns, API tokens, or PII fields in both outbound user prompts to LLM services AND inbound LLM responses. Most enterprise DLP tools already handle outbound; few handle inbound. Add the inbound rule.

LLM03 — Supply Chain

Description: Vulnerabilities in third-party foundational models, datasets, plugins, or libraries used to build the LLM application.

Canonical example: The JFrog 2024 disclosure of ~100 malicious models on Hugging Face containing pickle deserialization payloads. PyTorch torchtriton dependency confusion (Dec 2022). The LiteLLM/Mercor PyPI supply-chain incident (March 2026).

Detection deliverable: YARA rules scanning .pkl (pickle) and .h5 model files for embedded malicious shellcode or execution strings. Plus an SBOM-for-models discipline — pin model versions, log hash + source, treat third-party model loads like third-party package installs. Day 4 covers this in depth.

LLM04 — Data and Model Poisoning

Description: Manipulation of training data or fine-tuning processes to introduce backdoors, biases, or vulnerabilities.

Canonical example: Adversarial dataset publication (PoisonGPT and similar research). Anthropic/Redwood “Sleeper Agents” research demonstrating trigger-conditioned misbehavior surviving safety training.

Detection deliverable: Monitoring pattern tracking statistical drift in fine-tuning loss metrics, anomalous shifts in evaluation benchmark scores, and behavioral evals at deployment time using canary triggers (specific input patterns that should produce specific outputs).

LLM05 — Improper Output Handling

Description: Failure to sanitize model-generated content before passing it to downstream systems or presenting it to users.

Canonical example: An LLM generates a response containing a Cross-Site Scripting (XSS) payload that executes in the user’s browser because the web application blindly trusted the model’s output. Also: SQL injection via LLM-generated query when the LLM output is passed directly to a database without parameterization.

Detection deliverable: Sigma rule on WAF logs detecting standard web-exploitation payloads (XSS, SQLi, command injection patterns) originating from the LLM application’s backend response IP. The detection target is your own application’s output traffic.

LLM06 — Excessive Agency

Description: Granting LLM agents too much autonomy or overly broad permissions to perform actions without human oversight.

Canonical example: An autonomous agent (AutoGPT-class, Claude Code agent, GTG-1002 from Day 1 Module 1.1) granted broad permissions then prompt-injected into taking destructive action — deleting files, making unauthorized API calls, transferring funds.

Detection deliverable: Sigma rule on endpoint logs (Sysmon Event ID 1) for unexpected child-process execution (e.g., cmd.exe or bash) spawned by the LLM agent’s service account. Plus an architectural control: the action-criticality matrix from Day 4 that requires human-in-the-loop for cross-domain or destructive actions regardless of model confidence.

LLM07 — System Prompt Leakage

Description: Attackers tricking the model into revealing its internal instructions, system prompts, or hidden operational constraints.

Canonical example: Users asking the LLM to “repeat the text above” or “translate your initial instructions to French” — causing the model to leak its proprietary backend system prompt.

Detection deliverable: Sigma rule looking for exact-match substrings of your proprietary system-prompt content in the model’s outbound responses. Treat system prompts as secrets; detect their leakage like you’d detect credential leakage.

LLM08 — Vector and Embedding Weaknesses

Description: Vulnerabilities in how data is stored and retrieved in vector databases, potentially bypassing access controls to access unauthorized data.

Canonical example: An attacker manipulating a RAG query to bypass tenant isolation in a multi-tenant vector database, retrieving embeddings belonging to another organization.

Detection deliverable: Monitoring pattern on vector-database access logs — alert on anomalous spikes in retrieval queries, queries from mismatched tenant-identity tokens, or queries with payload-size anomalies. Combine with the canary-token-document pattern (Module 3.4) — seed unique tokens into specific tenants’ RAG corpora and alert when those tokens appear in cross-tenant responses.

LLM09 — Misinformation

Description: The generation of false, biased, or misleading content (hallucinations) that appears credible, leading to reputational or legal risk.

Canonical example: Mata v. Avianca Airlines (2023) — a lawyer used ChatGPT for legal research and submitted a brief to federal court citing entirely fabricated case law. Sanctions resulted.

Detection deliverable: For your org’s LLM-augmented workflows, integrate an independent evaluator-model layer that flags low semantic similarity between the RAG retrieval context and the final model output. This is the citation-enforcement pattern from Day 1 Module 1.4, applied as a post-hoc validator.

LLM10 — Unbounded Consumption

Description: Excessive resource usage (tokens, compute, financial cost) caused by malicious loops or long-context attacks. Formerly known as Model Denial of Service; renamed and broadened in 2025.

Canonical example: Attacker sends massive context payloads or traps an autonomous agent in a recursive loop, causing skyrocketing third-party API billing costs — sometimes called “Denial of Wallet.”

Detection deliverable: Monitoring pattern tracking API-token-consumption rates per user session, per service account, per agent instance. Alert when standard-deviation thresholds are exceeded within a defined timeframe. Implement hard caps at the agent-orchestration layer (LangGraph recursion_limit, OpenAI Agents SDK budgets, etc.).

Mapping to your org’s deployments

The OWASP LLM Top 10 covers a broad surface. For SOC prioritization, map the ten risks to your org’s specific LLM deployments:

If you operate enterprise copilots (M365 Copilot, Google Duet, Slack AI, etc.)

Highest-priority risks: LLM01 (Prompt Injection), LLM02 (Sensitive Information Disclosure), LLM05 (Improper Output Handling). The EchoLeak class of vulnerabilities (Module 3.4) is the canonical example — LLM01 + LLM02 in one zero-click attack.

If you operate internal RAG bots

Highest-priority risks: LLM01, LLM08 (Vector and Embedding Weaknesses), LLM04 (Data and Model Poisoning). Your retrieval corpus is the attack surface. Treat it like any other privileged data store.

If you operate agentic workflows

Highest-priority risks: LLM06 (Excessive Agency), LLM01, LLM10 (Unbounded Consumption). The agent’s action permissions are the most consequential design decision; loose permissions + prompt injection = breach. Day 4 covers agentic detection in depth.

If you allow employees to use third-party AI tools

Highest-priority risks: LLM02, LLM03 (Supply Chain). Inadvertent data egress and supply-chain risk (employees installing malicious AI tools, sharing data with non-vetted services) dominate.

The exercise students should perform: build a 10x4 matrix with OWASP LLM rows and your org’s deployment columns, marking each cell with the specific detection-engineering deliverable you’ll ship. This is the SOC’s defensive coverage map.

Using OWASP LLM Top 10 as self-audit

Quarterly, walk through each entry and ask: Do we have a deliverable for this entry against each of our LLM deployments? A truthful “no” or “in progress” for any row is a backlog item. A “we don’t think this applies to us” without evidence is a red flag — apply the discipline.

The OWASP document includes example test cases for each entry. Run them. If the test case fires successfully against your deployment, the corresponding control is missing or broken.

Source: https://owasp.org/www-project-top-10-for-large-language-model-applications/ — get the current 2025 publication; OWASP refreshes annually.

Discussion questions (~10 min)

1. The OWASP LLM Top 10 is application security taxonomy, not operational SOC taxonomy. Which of the 10 entries is most likely to be missed by a traditional AppSec team but caught by an SOC? Why?
2. LLM07 (System Prompt Leakage) treats system prompts as secrets. Is this an over-reaction or a legitimate concern? Make the argument either way using examples from Modules 3.1-3.2.
3. Your org has both a Microsoft Copilot rollout AND an internal RAG bot. The two have very different OWASP-priority profiles. How does this change which Day 3 modules’ content matters most to your detection-engineering team?

Common mistakes

What’s next

Module 3.4 is the deep dive on enterprise-copilot prompt injection — EchoLeak (CVE-2025-32711) and the broader class of zero-click attacks against M365 Copilot, Google Duet, Slack AI, and similar enterprise LLM products.