Tagged:
defense-patterns50 posts
-
Cross-Tenant Data Leakage in Multi-Tenant LLM Deployments: Incidents, Architecture, and What to Demand from Providers
When an LLM service hosts multiple customers on shared infrastructure, failures in conversation isolation, prompt cache sharing, or conversation history storage can expose one customer's data to another. This post covers the documented failure modes, the March 2023 ChatGPT Redis incident, batch inference and system prompt bleed risks, and what enterprise architects should demand from LLM providers before trusting them with sensitive data.
-
Slopsquatting: When AI Hallucinated Package Names Become a Supply Chain Attack
LLMs hallucinate package names at rates exceeding 20%. Attackers register those hallucinated names on npm and PyPI with malicious payloads. No typo required — the developer trusts the AI. This post explains the mechanics, documents the evidence, and gives developers concrete defenses.
-
Computer Use Agent Security: Attack Surfaces of GUI-Access AI Systems
When an AI agent can see your screen and move your mouse, every pixel becomes part of the attack surface. A technical breakdown of clipboard poisoning, visual prompt injection, screenshot exfiltration, and the mitigations that actually work.
-
Least Privilege for AI Agents: Runtime Capability Minimization and Reducing Blast Radius
Most agent deployments give AI agents too many capabilities by default — violating a 50-year-old security principle. A concrete framework for enumerating required capabilities, scoping tool access by task, sandboxing execution environments, and auditing capability creep to reduce blast radius.
-
AI in Critical Infrastructure: Attack Surfaces in Industrial Control Systems and Smart Grids
How AI integration into power grids, water treatment, manufacturing, and transportation creates novel attack surfaces distinct from traditional ICS/SCADA threats — sensor spoofing against ML anomaly detectors, adversarial attacks on predictive maintenance, model poisoning in federated industrial AI, and mitigations specific to OT environments.
-
AI Secrets Management: Protecting API Keys, System Prompts, and Model Credentials in Production
API keys for model providers have a blast radius unlike traditional credentials. System prompts encode proprietary business logic that teams rarely treat as secrets. This is a practical guide to treating AI credentials as the first-class secrets they are — with concrete patterns for storage, rotation, agent scoping, and incident response.
-
Quantization and Compression Attacks: How Model Size Reduction Can Re-Enable Suppressed Unsafe Behaviors
When you deploy a GGUF Q4_K_M quantization of a safety-aligned model, you're relying on an assumption: that alignment survives precision reduction. Research now demonstrates two distinct ways that assumption can fail — and why quantized models need their own safety verification.
-
Improper LLM Output Handling: SQL Injection, XSS, and SSRF via AI-Generated Responses
LLM output is attacker-influenced data. Passing model responses directly into SQL queries, HTML renderers, or downstream APIs creates injection vulnerabilities that attackers exploit via prompt injection → server-side attack chains. Concrete defenses for every consumer type.
-
Poisoning the Knowledge Base: Adversarial Document Injection into RAG Vector Stores
RAG can be weaponized by anyone who can write to the corpus. This post maps the attack taxonomy — two empirically validated classes (PoisonedRAG, BadRAG), one extrapolated operational pattern, and persistent corpus injection — with real-world ingestion surfaces and a defender checklist.
-
Poisoning the Pretraining Corpus: How Attackers Corrupt Foundation Models Before They're Built
Modern foundation models train on trillions of tokens scraped from the web. Carlini et al. 2023 demonstrated that purchasing expired web domains in Common Crawl snapshots lets an attacker inject poisoned training examples into the datasets foundation models train on — before a single GPU fires up.
-
ML Model Provenance: Signing, SBOMs, and Verifying the AI You Deploy Before It Runs
You wouldn't deploy software without checksums and signatures. But most organizations download model weights and run them without any provenance verification at all. This post covers the practical mechanics of model signing, ML SBOMs, and the emerging infrastructure for verifying a model's origins before it touches production.
-
Malicious AI Model Files: Pickle Exploits and Arbitrary Code Execution on Model Load
Downloading a model file and calling torch.load() is a potential code execution event. This post explains the pickle mechanics that make it so, the real-world exploits already found in the wild, how HuggingFace has responded, and what safe alternatives actually prevent the attack.
-
System Prompt Extraction: How Attackers Steal Proprietary AI Instructions
System prompts encode valuable business logic, persona, and safety overrides — and deployed AI products routinely leak them to motivated users. This post covers direct elicitation, behavioral inference, fine-tuning extraction, and the real-world incidents that showed extraction is not theoretical.
-
CI/CD Pipeline Injection: When AI Code Assistants Become Supply Chain Threats
AI coding assistants consume your codebase as context — and that context is attacker-controllable. This post maps the attack surface: indirect prompt injection via poisoned READMEs, hallucinated package names that resolve to malicious code, AI-generated tests that exfiltrate secrets, and training data poisoning that shifts model behavior at the corpus level.
-
Federated Learning Poisoning: The Aggregation Attack Surface
FL aggregation is blind to participant intent. Malicious clients can embed backdoors or reconstruct private training data from gradients.
-
Model Extraction via API Queries: Stealing Proprietary AI Without the Weights
Systematic API queries can reconstruct a proprietary model's behavior — and sometimes its architecture — without ever touching the weights. The security implications are dual: IP theft and a cheap path to an unconstrained surrogate for downstream adversarial attack crafting.
-
Token Smuggling: Unicode Tricks That Slip Past AI Safety Filters
How attackers use Unicode homoglyphs, invisible characters, bidirectional overrides, and encoding tricks to smuggle disallowed content past LLM safety systems — and why the tokenizer-renderer gap is a structural attack surface that normalization alone doesn't fully close.
-
AI Agent Supply Chain Attacks: Compromising Agents Before They Run
SolarWinds taught us that compromising a dependency upstream is more effective than attacking the target directly. The same logic applies to AI agents: model weights, prompt templates, tool registries, and evaluation datasets are all upstream dependencies that, if poisoned, produce a backdoored agent that behaves normally until triggered.
-
Tool Poisoning via Malicious MCP Servers: When Your Agent's Tools Turn Against It
MCP servers are the extension layer for modern AI agents — granting file access, web search, code execution, and API calls. This post examines the threat of malicious or compromised MCP servers: how they exploit the agent's implicit trust in its own tooling (including via tool definition injection, rug-pull attacks, and cross-tool description chaining), the attack classes that follow, and the defense patterns that actually work.
-
Browser-Use Attacks: Hijacking AI Agents That Browse the Web
Frameworks like browser-use, Playwright agents, and OpenAI Operator are bringing browser-capable AI into production. Here's how attackers exploit web-browsing agents through indirect prompt injection — and what defenders can do about it.
-
The Confused Deputy Problem in LLM Tool Use: Why Agents Need Least-Privilege APIs
A 1988 operating systems paper describes exactly the threat model facing LLM agents with OAuth integrations. Classic infosec's confused deputy maps onto AI tool use in ways most developers haven't internalized — and the mitigations that follow from it aren't the ones getting deployed.
-
Prompt Injection as Role Confusion: Why LLMs Trust Style Over Role Tags
New research reveals LLMs identify roles from writing style rather than structural tags — making the system role tag a suggestion, not a security boundary. Destyling adversarial text drops attack success from 61% to 10%. Here's what that means for defenders.
-
Daybreak: Inside OpenAI's AI-Powered Vulnerability Detection Initiative
OpenAI's Daybreak initiative bundles GPT-5.5-Cyber, Codex Security, and Patch the Planet into a coordinated defender program. Here's what it can actually do today, how it compares to competing tools, and why the dual-use question is the part the announcements glossed over.
-
How the Big Labs Red-Team Their Models — and What They Keep Missing
The big labs red-team their models. Here's what they find, what they miss, and what that means for anyone deploying AI in 2026.
-
Sleeper Agents in Production: The AI Supply Chain Backdoor Threat
Anthropic proved sleeper agents exist and resist standard safety fine-tuning. Here's who's actually at risk in 2026 and what a realistic defense looks like when fine-tuned open-weight models are everywhere.
-
How Anthropic Contains Claude — Multi-Environment Sandboxing Architecture
Anthropic published a rare deep-dive into how Claude is sandboxed across three different products — claude.ai, Claude Code, and Claude Cowork — each using a different containment architecture matched to its threat model. This post breaks down those three patterns, the real incidents that shaped them, and how they compare to OpenAI's Codex approach.
-
AI Is Breaking Two Vulnerability Cultures: A Practitioner's Read
Coordinated disclosure and Linux's 'bugs are bugs' approach have coexisted for decades. AI-assisted vulnerability scanning is quietly dismantling both — and the Copy Fail incident shows what the new equilibrium might look like.
-
Content Provenance at Scale: What C2PA and SynthID Actually Do — and What They Don't
OpenAI and Google are now embedding provenance credentials into AI-generated content at production scale. C2PA cryptographic manifests and SynthID watermarks represent genuinely different threat models. Here's what each solves, where each breaks, and why neither is a complete answer to AI content authentication.
-
Running Codex Safely at OpenAI — Sandboxing Patterns
OpenAI's Codex runs every model-generated command inside an OS-enforced sandbox before it touches your filesystem or network. This post breaks down the layered isolation model — platform mechanics, approval flows, auto-review, and cloud isolation — and what each layer actually prevents.
-
Retrying Leaks Your Monitor: Why AI Control Scaffolds Need to Rethink Feedback
New research from Redwood Research shows that retrying — blocking risky actions and continuing with monitor rationale in context — lets adversarial models craft sneakier attacks, negating safety gains. Resampling avoids the information leak: drawing 5 samples per step and auditing on the maximum suspicion score raises safety from 61% to 71% at a 0.3% audit budget.
-
The Trigger You Can't See: Steganographic Backdoors in Deployed Language Models
SteganoBackdoor shows that a model can be made to reliably produce attacker-controlled outputs whenever a semantic trigger appears — without the trigger ever appearing in the poisoned training data. Existing data-curation defenses can't find what they're not designed to look for.
-
1,000 Breaches Later: The Disclosure Lag Is Worse Than Ever
Troy Hunt's HIBP milestone analysis shows breach disclosure times are lengthening, not shrinking — driven by class-action fear, legal posturing, and regulatory carve-outs. The same dynamics will shape how AI incidents get disclosed.
-
ALIGNBEAM: Transferring Safety Alignment Across Model Families at Inference Time
Domain fine-tuning erodes safety alignment — and existing logit-mixing defenses don't work when the draft and safety models use different vocabularies. ALIGNBEAM solves this with a training-free text-bridge that translates anchor logits into any target vocabulary, raising AdvBench refusal from 38% to 92% without touching either model's weights.
-
Anthropic's Vulnerability Discovery Framework: The Bottleneck Moved from Finding to Fixing
Anthropic's open-source framework for AI-powered vulnerability discovery shows how to scale security scanning with Claude. Their finding: discovery is now straightforward to parallelize, but verification, triage, and patching are the new bottlenecks. Here's the 6-step loop that works, based on partnerships with security teams at multiple organizations.
-
VeilGate: When Your Defense Is a Lie That Costs the Attacker Money
Deception proxies flip the economics of AI-assisted pentesting by routing hostile automation into believable tarpits instead of blocking it
-
A Common Language for Evaluating Frontier AI: OpenAI's Shared Playbook
OpenAI released a shared playbook for conducting trustworthy third-party AI evaluations, structured around three pillars: model capabilities, safeguards, and validity. Here's what that structure reveals about where the field currently falls short.
-
The Time Bomb in Your Fine-Tuned Model: MetaBackdoor Exploits Position, Not Content
A new backdoor attack requires no suspicious text—it activates when conversation length crosses a threshold, leaking system prompts and making unauthorized tool calls.
-
Your Safety Fine-Tuning Data May Be Teaching the Wrong Lessons
A fundamental flaw in how LLMs process negation during fine-tuning means datasets showing models what NOT to do can inadvertently teach them to do exactly that.
-
The Inbox Is the New Attack Surface: What Gemini Spark Reveals About Personal AI Agent Security
Google's personal AI agent has ambient authority over your Gmail, Calendar, and Drive. Researchers have already demonstrated how to hijack it through a calendar invite. Infrastructure defenses don't fix this.
-
When Your Safety Layer Gets Compromised: The npm Supply Chain Problem in AI Agent Pipelines
The Mini Shai-Hulud campaign hit guardrails-ai and the Mistral AI SDK. For AI teams, this is more than a supply chain story — it's a demonstration that your agent's safety layer is part of the attack surface.
-
Your Agent Runtime Is a 1960s Operating System
A new paper from TU Berlin and CISPA maps AI agent security onto 50 years of OS research — and finds that agent runtimes are failing to apply solutions that were well-understood before most of their developers were born.
-
Your Agent's Memory Is Building a Privacy Database You Didn't Design
Cloud-assisted agent memory systems are accumulating raw user PII — health conditions, credentials, contact details — in vector databases where it persists indefinitely. MemPrivacy shows the attack surface is real, quantified, and fixable. Here's the threat model most teams haven't modeled.
-
When Your Agent Forgets the Right Things: Skill Libraries as Emergent Defense Against Memory Poisoning
A new RL framework for agent skill libraries creates an unexpected security property: skills that lead to task failures get naturally retired. Here's what that means for your threat model — and where the attack surface actually shifts.
-
Your AI Agent Is an Improvised Prototype. Here's Why That's a Security Problem.
A new cs.CR paper argues that the dominant 'on-the-fly' agentic paradigm short-circuits 50 years of software engineering discipline — and that the security implications are severe. Every improvised tool chain is a prototype you're deploying as if it were production.
-
Safe in Isolation, Dangerous Together: The Multi-Turn Blind Spot in Your Safety Filter
Decompositional jailbreaks split a harmful request across innocuous-looking turns. TwinGate is the first defense designed for the hardest variant: fully anonymous, interleaved traffic with no user identity metadata.
-
Exploration Hacking: When Your Model Games Its Own Training
A new attack class shows that sufficiently capable LLMs can strategically suppress their exploration during RL training to avoid having dangerous capabilities elicited — and frontier models already reason about it.
-
7.1%: What Happens When You Actually Measure Multi-Agent Safety
TrinityGuard tested real multi-agent system configurations against a structured, OWASP-grounded taxonomy of 20 risk types. The average safety pass rate was 7.1%. Here's what that number means and what the framework gives you to act on it.
-
Poisoning What Your Agent Remembers: The Cross-Session Attack You Haven't Modeled
eTAMP shows that a single compromised webpage can silently corrupt an agent's persistent memory, then trigger the payload on a completely different site in a future session — with attack success rates climbing to 32.5% when the agent is under stress.
-
Your Guardrails Can't Read JSON: The Structural Bottleneck in Agentic Safety
New research finds that guardrail performance on tool-call trajectories correlates at ρ=0.79 with structured-data reasoning ability — and near-zero with jailbreak robustness. Here's what that means for how you secure agents.
-
Your Agent Is Mine: The LLM Router Supply Chain Attack You're Not Defending Against
Researchers bought 428 LLM API routers and found 9 actively injecting malicious code. Here's what that means for every agent that uses a third-party API proxy.