Tagged:
prompt-injection43 posts
-
Prompt Cache Timing Attacks: Side-Channel Leakage in LLM API Infrastructure
Major LLM providers cache prompt prefixes to reduce latency and cost — but shared cache infrastructure creates a timing side-channel. An attacker with access to a shared API key or multi-tenant deployment can measure whether a specific prefix is cached, leaking information about co-tenant system prompts and conversation context within the same cache namespace.
-
Computer Use Agent Security: Attack Surfaces of GUI-Access AI Systems
When an AI agent can see your screen and move your mouse, every pixel becomes part of the attack surface. A technical breakdown of clipboard poisoning, visual prompt injection, screenshot exfiltration, and the mitigations that actually work.
-
Prompt Injection in Email, Calendars, and Productivity Tools: The Enterprise AI Copilot Attack Surface
Microsoft 365 Copilot, Gemini in Workspace, and similar AI assistants have read-write access to your entire mailbox, calendar, and document corpus. A single malicious email can hijack them to exfiltrate your inbox history, forward confidential documents, or impersonate you in outgoing messages — no exploit required.
-
Improper LLM Output Handling: SQL Injection, XSS, and SSRF via AI-Generated Responses
LLM output is attacker-influenced data. Passing model responses directly into SQL queries, HTML renderers, or downstream APIs creates injection vulnerabilities that attackers exploit via prompt injection → server-side attack chains. Concrete defenses for every consumer type.
-
Multimodal Jailbreaking: How Attackers Use Images to Bypass Text Safety Filters
In many multimodal deployments, text safety classifiers operate on tokens — not image pixels. Attackers exploit this gap by embedding harmful instructions in rendered typography, screenshots, and contextually manipulative images — bypassing filters that block the same content in plain text.
-
Defending Against Prompt Injection: Privilege Separation, Structured Outputs, and the Limits of Current Defenses
No single control stops prompt injection — but layering four concrete patterns (privilege separation, structured outputs, instruction hierarchy, spotlighting untrusted content) gives defenders a workable stack. This post synthesizes what actually works, and is honest about what remains unsolved.
-
Reasoning Model Security: Attacks on Chain-of-Thought and Extended Thinking
Reasoning models (o3, o4-mini, DeepSeek-R1, Claude with extended thinking) expose a distinct attack surface that standard LLM defenses don't fully address — attackers can manipulate the reasoning process itself, not just the inputs or outputs.
-
Poisoning the Knowledge Base: Adversarial Document Injection into RAG Vector Stores
RAG can be weaponized by anyone who can write to the corpus. This post maps the attack taxonomy — two empirically validated classes (PoisonedRAG, BadRAG), one extrapolated operational pattern, and persistent corpus injection — with real-world ingestion surfaces and a defender checklist.
-
AI Incident Response: A Practitioner's Playbook for When Your AI System Is Compromised
Detection signals, containment options, evidence preservation, and recovery procedures for AI-specific security incidents — the operational complement to attack coverage. NIST IR lifecycle applied to prompt injection, model backdoors, data poisoning, and adversarial input attacks.
-
LLM Guardrails in Practice: A Decision Guide to Runtime Input/Output Filtering Tools
LlamaGuard, Azure Prompt Shield, PromptGuard, NeMo Guardrails, Guardrails AI, and Presidio — how each works and when to deploy it.
-
Attacking the Judge: Adversarial Manipulation of LLM-as-a-Judge Evaluation Systems
LLM-as-a-Judge systems are now infrastructure for RLHF, automated red-teaming, and production quality gates. This post covers the emerging attack surface: how adversaries can manipulate evaluation systems by controlling the content being judged, and what that means for AI pipelines.
-
System Prompt Extraction: How Attackers Steal Proprietary AI Instructions
System prompts encode valuable business logic, persona, and safety overrides — and deployed AI products routinely leak them to motivated users. This post covers direct elicitation, behavioral inference, fine-tuning extraction, and the real-world incidents that showed extraction is not theoretical.
-
Agent Loop Hijacking: How Resource Exhaustion and Infinite Reasoning Loops Become Attack Primitives
Most agent security thinking focuses on what an agent does — exfiltration, injection, privilege escalation. Equally dangerous is forcing an agent to do nothing useful, or bankrupting the operator through runaway compute. This post breaks down loop hijacking and resource exhaustion as deliberate attack primitives.
-
RAG Privacy Attacks: How Retrieval-Augmented Generation Pipelines Leak Private Documents
RAG makes LLMs accurate by indexing private documents — but the retrieval pipeline introduces a new attack surface. Adversarial queries can extract document chunks, embedding inversions recover original text, and multi-tenant isolation can fail in ways that expose documents across access boundaries.
-
CI/CD Pipeline Injection: When AI Code Assistants Become Supply Chain Threats
AI coding assistants consume your codebase as context — and that context is attacker-controllable. This post maps the attack surface: indirect prompt injection via poisoned READMEs, hallucinated package names that resolve to malicious code, AI-generated tests that exfiltrate secrets, and training data poisoning that shifts model behavior at the corpus level.
-
Prompt Injection in Long-Context Windows: When More Context Means More Attack Surface
Extended context windows (128K–1M tokens) introduce an underappreciated prompt injection attack surface: injected instructions buried deep in retrieved documents, conversation histories, or tool outputs can override system prompts or hijack reasoning far from the visible top of the context. This post maps the threat, explains why attention geography matters for defenders, and offers a practical mitigation checklist.
-
Shadow Prompting: How Hidden System Instructions Hijack AI Behavior
Hidden system-level instructions can silently override how an AI application behaves — without the user ever seeing the manipulation. This post explores what makes shadow prompting possible, how it differs from ordinary prompt injection, and what architectural choices actually reduce the risk.
-
Token Smuggling: Unicode Tricks That Slip Past AI Safety Filters
How attackers use Unicode homoglyphs, invisible characters, bidirectional overrides, and encoding tricks to smuggle disallowed content past LLM safety systems — and why the tokenizer-renderer gap is a structural attack surface that normalization alone doesn't fully close.
-
Adversarial Attacks on Vision-Language Models: Pixels as Injection Vectors
Gradient-crafted image perturbations can override VLM safety guardrails, inject attacker-defined instructions, and trigger tool calls in agentic pipelines. A survey of the empirically confirmed attack landscape and practical defenses.
-
Adversarial Prompt Caching: Timing Attacks and Injection via Shared KV Caches
As LLM providers roll out shared prefix caching to cut inference costs, a new attack surface emerges: cache-timing side channels and cross-tenant injection vectors at the inference infrastructure layer.
-
Exfiltration via Agent Side Channels: How AI Agents Leak Sensitive Data Indirectly
When you can't send the data directly, you encode it in everything else. A taxonomy of side-channel exfiltration paths available to compromised AI agents — and why most DLP monitoring misses every one of them.
-
Tool Poisoning via Malicious MCP Servers: When Your Agent's Tools Turn Against It
MCP servers are the extension layer for modern AI agents — granting file access, web search, code execution, and API calls. This post examines the threat of malicious or compromised MCP servers: how they exploit the agent's implicit trust in its own tooling (including via tool definition injection, rug-pull attacks, and cross-tool description chaining), the attack classes that follow, and the defense patterns that actually work.
-
Browser-Use Attacks: Hijacking AI Agents That Browse the Web
Frameworks like browser-use, Playwright agents, and OpenAI Operator are bringing browser-capable AI into production. Here's how attackers exploit web-browsing agents through indirect prompt injection — and what defenders can do about it.
-
The Confused Deputy Problem in LLM Tool Use: Why Agents Need Least-Privilege APIs
A 1988 operating systems paper describes exactly the threat model facing LLM agents with OAuth integrations. Classic infosec's confused deputy maps onto AI tool use in ways most developers haven't internalized — and the mitigations that follow from it aren't the ones getting deployed.
-
Prompt Injection as Role Confusion: Why LLMs Trust Style Over Role Tags
New research reveals LLMs identify roles from writing style rather than structural tags — making the system role tag a suggestion, not a security boundary. Destyling adversarial text drops attack success from 61% to 10%. Here's what that means for defenders.
-
AI Worms: How Self-Replicating Attacks Spread Through Multi-Agent Pipelines
Morris-II (Cohen, Bitton & Nassi, 2024) demonstrated that malicious prompts can self-replicate across GenAI ecosystems, spreading to new agents without requiring the target to interact with the malicious content. Here's what makes LLM pipelines worm-able — and what defenders can do about it.
-
Poisoning the Well: Memory and RAG Attacks Against Long-Context AI Systems
As AI agents gain persistent memory through vector stores and RAG pipelines, a new attack surface emerges: injecting malicious content into the agent's long-term knowledge to control future behavior. This post maps the taxonomy of memory poisoning attacks — from retrieval manipulation to cross-session contamination — and what defenders need to audit before production.
-
Defense-in-Depth for AI Agents — A Security Architect's Stack
Securing an AI agent isn't one control — it's a stack. A structured guide to layering security controls across the full agent architecture, from model selection through orchestration, tool execution, and output validation.
-
Indirect Prompt Injection Against Production Systems: A Survey of Documented Disclosures
Five researcher-disclosed proof-of-concept exploits against production AI systems (2023–2024) reveal recurring structural patterns that defenders can act on today: from Bing Chat web-content sideloading to Microsoft 365 Copilot email exfiltration chains.
-
MCP Security: The New Attack Surface for AI Tool Protocols
MCP is to AI agents what HTTP is to browsers — and it has the same early-web security baggage. Here's what developers and security engineers need to audit before shipping agent infrastructure.
-
OWASP Top 10 for AI Agents, Part 1: The Three Vulnerabilities That Break Agent Trust
Agents aren't just chatbots with tools attached. They're autonomous systems that read, write, call APIs, and act with your credentials. This series builds an agent-specific security checklist from first principles, starting with the three vulnerability classes that the OWASP LLM Top 10 2025 elevated: Prompt Injection, Excessive Agency, and Insecure Tool Chaining.
-
FinHarness: Multi-tier Inline Safety Harness for Finance LLM Agents
FinHarness wraps finance agents end-to-end with a Query Monitor, Tool Monitor, and adaptive Cascade judge — cutting attack success rate from 38.3% to 15.0% while keeping benign approval at 39.3%.
-
Five Lines of Injection: How Microsoft Copilot Cowork Exfiltrates Pre-Authenticated File Links Without Approval
PromptArmor demonstrated that five malicious lines hidden in a Copilot Cowork Skills file can exfiltrate pre-authenticated OneDrive and SharePoint download links — no-login URLs granting instant file access — without any user approval. A 5-for-5 success rate, model-agnostic, and no patch available. The attack exposes a structural flaw in agent approval design: classifying actions by destination rather than content.
-
Your Spreadsheet Is the Attack Surface: ChatGPT for Google Sheets Data Exfiltration
A single hidden prompt in an imported sheet can silently exfiltrate your entire Google Drive workbook collection, replace the ChatGPT sidebar with an attacker-controlled phishing interface, and bypass the extension's own human-approval safety setting. PromptArmor's analysis of the ChatGPT for Google Sheets add-on shows how LLM extensions inherit every permission you've granted — and hand them to whoever controls the data you import.
-
One Prompt, One High-Profile Instagram Account: How Meta's Support Bot Became an Exploit
Hackers bypassed Meta's AI support chatbot with a single text prompt to take over high-profile Instagram accounts — including the Barack Obama White House account. The bot had account recovery powers, no human escalation path, and failed at its most critical function: distinguishing between the account owner and an attacker who knew the right words.
-
The Inbox Is the New Attack Surface: What Gemini Spark Reveals About Personal AI Agent Security
Google's personal AI agent has ambient authority over your Gmail, Calendar, and Drive. Researchers have already demonstrated how to hijack it through a calendar invite. Infrastructure defenses don't fix this.
-
Safe in Isolation, Dangerous Together: The Multi-Turn Blind Spot in Your Safety Filter
Decompositional jailbreaks split a harmful request across innocuous-looking turns. TwinGate is the first defense designed for the hardest variant: fully anonymous, interleaved traffic with no user identity metadata.
-
Poisoning What Your Agent Remembers: The Cross-Session Attack You Haven't Modeled
eTAMP shows that a single compromised webpage can silently corrupt an agent's persistent memory, then trigger the payload on a completely different site in a future session — with attack success rates climbing to 32.5% when the agent is under stress.
-
When AI Agents Talk in Embeddings, Text-Level Safety Filters Go Blind
RecursiveMAS replaces inter-agent text communication with latent-space embeddings for efficiency. The security consequence: an entirely new attack surface — latent-space injection — where adversarial representations propagate between agents with no text transcript, no content filter, and no audit trail.
-
Safe Agents, Unsafe Systems: The Non-Compositionality Problem in Multi-Agent Security
A 24-author paper from Oxford, CMU, MIT, and the Turing Institute argues that individually safe AI agents can compose into unsafe systems — and that securing each agent in isolation misses the point entirely.
-
What Red-Teaming Misses When Agents Talk to Each Other
Microsoft Research red-teamed a live 100+ agent platform and found four attack classes — worms, amplification, trust capture, proxy chains — that only emerge at network scale. Single-agent benchmarks miss all of them.
-
Your Agent Is Mine: The LLM Router Supply Chain Attack You're Not Defending Against
Researchers bought 428 LLM API routers and found 9 actively injecting malicious code. Here's what that means for every agent that uses a third-party API proxy.
-
Three Papers, Three Attack Layers: Agent Security Gets Mapped
In one week, three independent research groups dissected the conversation, tool-use, and capability layers of AI agent systems. Here's what practitioners need to know.