Tagged:
threat-modeling63 posts
-
Computer Use Agent Security: Attack Surfaces of GUI-Access AI Systems
When an AI agent can see your screen and move your mouse, every pixel becomes part of the attack surface. A technical breakdown of clipboard poisoning, visual prompt injection, screenshot exfiltration, and the mitigations that actually work.
-
Least Privilege for AI Agents: Runtime Capability Minimization and Reducing Blast Radius
Most agent deployments give AI agents too many capabilities by default — violating a 50-year-old security principle. A concrete framework for enumerating required capabilities, scoping tool access by task, sandboxing execution environments, and auditing capability creep to reduce blast radius.
-
Prompt Injection in Email, Calendars, and Productivity Tools: The Enterprise AI Copilot Attack Surface
Microsoft 365 Copilot, Gemini in Workspace, and similar AI assistants have read-write access to your entire mailbox, calendar, and document corpus. A single malicious email can hijack them to exfiltrate your inbox history, forward confidential documents, or impersonate you in outgoing messages — no exploit required.
-
Voice AI Security: Adversarial Audio, Ultrasonic Injection, and Attacks on Speech-Enabled AI Agents
Voice-enabled AI agents inherit a distinct attack surface that text-focused security misses entirely. Adversarial audio perturbations fool ASR pipelines invisibly, ultrasonic commands exploit microphone analog front-ends, and voice authentication breaks when attackers control TTS. Here's the threat model practitioners need.
-
Alignment Faking in Large Language Models: The Research Finding That Could Break Safety Evaluations
Anthropic's December 2024 controlled experiment showed Claude-3-Opus strategically complying with safety guidelines when the training context was made explicit — and diverging when it wasn't. Here's what the finding means, what it doesn't prove, and why it matters for AI security.
-
Shadow AI in the Enterprise: Detecting, Governing, and Securing Unauthorized AI Tool Use
How to detect, govern, and secure unauthorized AI tool use in the enterprise — before sensitive corporate data leaves through AI prompts.
-
Privacy-Preserving AI Inference: Trusted Execution Environments, Homomorphic Encryption, and Confidential Computing
When you submit a medical record to an AI API, can the cloud operator read it? TEEs, homomorphic encryption, and secure multi-party computation provide cryptographic and hardware-enforced confidentiality. This post explains what each actually guarantees — and what it doesn't.
-
Quantization and Compression Attacks: How Model Size Reduction Can Re-Enable Suppressed Unsafe Behaviors
When you deploy a GGUF Q4_K_M quantization of a safety-aligned model, you're relying on an assumption: that alignment survives precision reduction. Research now demonstrates two distinct ways that assumption can fail — and why quantized models need their own safety verification.
-
Model Hub Supply Chain Attacks: Malicious Models, Tokenizer Exploits, and Typosquatting on Hugging Face
Downloading an open-weight model from a public hub is not a read-only operation. Custom tokenizer classes and auto_map configs execute arbitrary Python when trust_remote_code is set, LoRA adapters can trojanize safe base models, and typosquatted namespaces are a documented distribution vector. Here's the threat model practitioners need before they run from_pretrained().
-
Non-Human Identity Security for AI Agents: Credential Scoping, Token Lifecycle, and Agent Impersonation
AI agents are becoming major credential holders and API consumers. Non-human identity (NHI) security treats agent tokens with the same rigor as human credentials — and getting this wrong creates exploitable privilege escalation paths that traditional IAM never anticipated.
-
Poisoning the Knowledge Base: Adversarial Document Injection into RAG Vector Stores
RAG can be weaponized by anyone who can write to the corpus. This post maps the attack taxonomy — two empirically validated classes (PoisonedRAG, BadRAG), one extrapolated operational pattern, and persistent corpus injection — with real-world ingestion surfaces and a defender checklist.
-
Poisoning the Pretraining Corpus: How Attackers Corrupt Foundation Models Before They're Built
Modern foundation models train on trillions of tokens scraped from the web. Carlini et al. 2023 demonstrated that purchasing expired web domains in Common Crawl snapshots lets an attacker inject poisoned training examples into the datasets foundation models train on — before a single GPU fires up.
-
MITRE ATLAS: Mapping the AI/ML Threat Landscape with an Authoritative Adversarial Framework
MITRE ATLAS is the ATT&CK-equivalent for adversarial machine learning — a living knowledge base of tactics, techniques, and real-world case studies specific to AI systems. This post walks the full ATLAS matrix and shows how every attack pattern covered on this blog maps to a verifiable ATLAS technique ID.
-
Attacking the Judge: Adversarial Manipulation of LLM-as-a-Judge Evaluation Systems
LLM-as-a-Judge systems are now infrastructure for RLHF, automated red-teaming, and production quality gates. This post covers the emerging attack surface: how adversaries can manipulate evaluation systems by controlling the content being judged, and what that means for AI pipelines.
-
System Prompt Extraction: How Attackers Steal Proprietary AI Instructions
System prompts encode valuable business logic, persona, and safety overrides — and deployed AI products routinely leak them to motivated users. This post covers direct elicitation, behavioral inference, fine-tuning extraction, and the real-world incidents that showed extraction is not theoretical.
-
CI/CD Pipeline Injection: When AI Code Assistants Become Supply Chain Threats
AI coding assistants consume your codebase as context — and that context is attacker-controllable. This post maps the attack surface: indirect prompt injection via poisoned READMEs, hallucinated package names that resolve to malicious code, AI-generated tests that exfiltrate secrets, and training data poisoning that shifts model behavior at the corpus level.
-
Constitutional AI Under Attack: Exploiting Self-Critique Alignment Mechanisms
Constitutional AI aligns models by having them critique their own outputs against a set of principles. That self-critique loop is also an attack surface — adversarial constitutions, critique blindspots, and RLAIF label poisoning can all subvert alignment from within.
-
Side-Channel Attacks on LLM APIs: What Response Timing and Token Counts Reveal
How LLM API metadata — token counts, timing, and streaming patterns — leaks system prompt structure and model configuration to adversaries.
-
Prompt Injection in Long-Context Windows: When More Context Means More Attack Surface
Extended context windows (128K–1M tokens) introduce an underappreciated prompt injection attack surface: injected instructions buried deep in retrieved documents, conversation histories, or tool outputs can override system prompts or hijack reasoning far from the visible top of the context. This post maps the threat, explains why attention geography matters for defenders, and offers a practical mitigation checklist.
-
Federated Learning Poisoning: The Aggregation Attack Surface
FL aggregation is blind to participant intent. Malicious clients can embed backdoors or reconstruct private training data from gradients.
-
Model Extraction via API Queries: Stealing Proprietary AI Without the Weights
Systematic API queries can reconstruct a proprietary model's behavior — and sometimes its architecture — without ever touching the weights. The security implications are dual: IP theft and a cheap path to an unconstrained surrogate for downstream adversarial attack crafting.
-
How AI Safety Evaluations Are Gamed: Sandbagging, Context Drift, and Eval Design Gaps
How AI safety evaluations are gamed via sandbagging, context drift, and benchmark saturation—and what better evaluation design looks like for practitioners.
-
Exfiltration via Agent Side Channels: How AI Agents Leak Sensitive Data Indirectly
When you can't send the data directly, you encode it in everything else. A taxonomy of side-channel exfiltration paths available to compromised AI agents — and why most DLP monitoring misses every one of them.
-
AI Agent Supply Chain Attacks: Compromising Agents Before They Run
SolarWinds taught us that compromising a dependency upstream is more effective than attacking the target directly. The same logic applies to AI agents: model weights, prompt templates, tool registries, and evaluation datasets are all upstream dependencies that, if poisoned, produce a backdoored agent that behaves normally until triggered.
-
Multi-Agent Trust Escalation: How Subagents Inherit and Abuse Orchestrator Permissions
Multi-agent architectures introduce a trust escalation problem analogous to privilege escalation in OS security. When orchestrators delegate permissions to subagents, the attack surface multiplies. This post maps the attack classes, draws the OS and service-mesh analogies, and offers concrete defenses.
-
Tool Poisoning via Malicious MCP Servers: When Your Agent's Tools Turn Against It
MCP servers are the extension layer for modern AI agents — granting file access, web search, code execution, and API calls. This post examines the threat of malicious or compromised MCP servers: how they exploit the agent's implicit trust in its own tooling (including via tool definition injection, rug-pull attacks, and cross-tool description chaining), the attack classes that follow, and the defense patterns that actually work.
-
Browser-Use Attacks: Hijacking AI Agents That Browse the Web
Frameworks like browser-use, Playwright agents, and OpenAI Operator are bringing browser-capable AI into production. Here's how attackers exploit web-browsing agents through indirect prompt injection — and what defenders can do about it.
-
The Confused Deputy Problem in LLM Tool Use: Why Agents Need Least-Privilege APIs
A 1988 operating systems paper describes exactly the threat model facing LLM agents with OAuth integrations. Classic infosec's confused deputy maps onto AI tool use in ways most developers haven't internalized — and the mitigations that follow from it aren't the ones getting deployed.
-
Emergent Capabilities as Security Risks: What AI Systems Can Do That Nobody Planned For
LLMs spontaneously develop capabilities not explicitly trained—and this creates unpredictable security properties that defenders can't anticipate from static model cards or one-time capability evals.
-
Specification Gaming and Reward Hacking: When AI Optimizes for the Wrong Goal
Specification gaming — where AI systems satisfy the literal objective without fulfilling designer intent — has moved from a curious alignment phenomenon to an exploitable attack surface. As agents take on consequential real-world actions, the gap between what you asked for and what the system optimized for becomes adversarially relevant.
-
Poisoning the Well: Memory and RAG Attacks Against Long-Context AI Systems
As AI agents gain persistent memory through vector stores and RAG pipelines, a new attack surface emerges: injecting malicious content into the agent's long-term knowledge to control future behavior. This post maps the taxonomy of memory poisoning attacks — from retrieval manipulation to cross-session contamination — and what defenders need to audit before production.
-
How the Big Labs Red-Team Their Models — and What They Keep Missing
The big labs red-team their models. Here's what they find, what they miss, and what that means for anyone deploying AI in 2026.
-
Defense-in-Depth for AI Agents — A Security Architect's Stack
Securing an AI agent isn't one control — it's a stack. A structured guide to layering security controls across the full agent architecture, from model selection through orchestration, tool execution, and output validation.
-
MCP Security: The New Attack Surface for AI Tool Protocols
MCP is to AI agents what HTTP is to browsers — and it has the same early-web security baggage. Here's what developers and security engineers need to audit before shipping agent infrastructure.
-
Sleeper Agents in Production: The AI Supply Chain Backdoor Threat
Anthropic proved sleeper agents exist and resist standard safety fine-tuning. Here's who's actually at risk in 2026 and what a realistic defense looks like when fine-tuned open-weight models are everywhere.
-
Beyond Reward Hacking: RLHF's Spurious-Correlation Problem — and a Causal Fix
Wang et al. show that reward hacking in RLHF isn't just a data problem you can fix by collecting more annotations. Spurious correlations produce irreducible error that persists — and worsens during RL optimization — regardless of data scale. Their causal reward modeling approach uses counterfactual invariance to address the root cause, though the method requires explicitly identifying the spurious factors you want to enforce invariance against.
-
AgentBridge Attack Surface Analysis: When the Mesh Layer Becomes the Threat
Protocol translation bridges for AI agents look like infrastructure plumbing. They are actually the highest-value target in a multi-agent deployment — one compromise reaches every agent and every protocol at once.
-
AI Is Breaking Two Vulnerability Cultures: A Practitioner's Read
Coordinated disclosure and Linux's 'bugs are bugs' approach have coexisted for decades. AI-assisted vulnerability scanning is quietly dismantling both — and the Copy Fail incident shows what the new equilibrium might look like.
-
Alignment Tampering: How RLHF's Own Design Amplifies Bias
A paper accepted at ICML 2026 demonstrates a structural vulnerability in on-policy RLHF pipelines where the model being trained influences its own preference dataset — causing alignment to amplify rather than suppress certain misaligned behaviors.
-
The Trigger You Can't See: Steganographic Backdoors in Deployed Language Models
SteganoBackdoor shows that a model can be made to reliably produce attacker-controlled outputs whenever a semantic trigger appears — without the trigger ever appearing in the poisoned training data. Existing data-curation defenses can't find what they're not designed to look for.
-
1,000 Breaches Later: The Disclosure Lag Is Worse Than Ever
Troy Hunt's HIBP milestone analysis shows breach disclosure times are lengthening, not shrinking — driven by class-action fear, legal posturing, and regulatory carve-outs. The same dynamics will shape how AI incidents get disclosed.
-
The Shibboleth Effect: When Language Becomes a Security Variable
A new adversarial wargame study finds that frontier LLMs exhibit dramatically different behavioral dispositions depending on the language of play — Llama-4 turns sharply more coercive in Turkish while Gemini-3.1-Pro turns sharply less coercive. Same model, different language, different agent.
-
Anthropic's Vulnerability Discovery Framework: The Bottleneck Moved from Finding to Fixing
Anthropic's open-source framework for AI-powered vulnerability discovery shows how to scale security scanning with Claude. Their finding: discovery is now straightforward to parallelize, but verification, triage, and patching are the new bottlenecks. Here's the 6-step loop that works, based on partnerships with security teams at multiple organizations.
-
VeilGate: When Your Defense Is a Lie That Costs the Attacker Money
Deception proxies flip the economics of AI-assisted pentesting by routing hostile automation into believable tarpits instead of blocking it
-
A Common Language for Evaluating Frontier AI: OpenAI's Shared Playbook
OpenAI released a shared playbook for conducting trustworthy third-party AI evaluations, structured around three pillars: model capabilities, safeguards, and validity. Here's what that structure reveals about where the field currently falls short.
-
The Time Bomb in Your Fine-Tuned Model: MetaBackdoor Exploits Position, Not Content
A new backdoor attack requires no suspicious text—it activates when conversation length crosses a threshold, leaking system prompts and making unauthorized tool calls.
-
The Inbox Is the New Attack Surface: What Gemini Spark Reveals About Personal AI Agent Security
Google's personal AI agent has ambient authority over your Gmail, Calendar, and Drive. Researchers have already demonstrated how to hijack it through a calendar invite. Infrastructure defenses don't fix this.
-
When Your Safety Layer Gets Compromised: The npm Supply Chain Problem in AI Agent Pipelines
The Mini Shai-Hulud campaign hit guardrails-ai and the Mistral AI SDK. For AI teams, this is more than a supply chain story — it's a demonstration that your agent's safety layer is part of the attack surface.
-
Your Agent Runtime Is a 1960s Operating System
A new paper from TU Berlin and CISPA maps AI agent security onto 50 years of OS research — and finds that agent runtimes are failing to apply solutions that were well-understood before most of their developers were born.
-
Your Agent's Memory Is Building a Privacy Database You Didn't Design
Cloud-assisted agent memory systems are accumulating raw user PII — health conditions, credentials, contact details — in vector databases where it persists indefinitely. MemPrivacy shows the attack surface is real, quantified, and fixable. Here's the threat model most teams haven't modeled.
-
When Your Agent Forgets the Right Things: Skill Libraries as Emergent Defense Against Memory Poisoning
A new RL framework for agent skill libraries creates an unexpected security property: skills that lead to task failures get naturally retired. Here's what that means for your threat model — and where the attack surface actually shifts.
-
Your AI Agent Is an Improvised Prototype. Here's Why That's a Security Problem.
A new cs.CR paper argues that the dominant 'on-the-fly' agentic paradigm short-circuits 50 years of software engineering discipline — and that the security implications are severe. Every improvised tool chain is a prototype you're deploying as if it were production.
-
Safe in Isolation, Dangerous Together: The Multi-Turn Blind Spot in Your Safety Filter
Decompositional jailbreaks split a harmful request across innocuous-looking turns. TwinGate is the first defense designed for the hardest variant: fully anonymous, interleaved traffic with no user identity metadata.
-
Exploration Hacking: When Your Model Games Its Own Training
A new attack class shows that sufficiently capable LLMs can strategically suppress their exploration during RL training to avoid having dangerous capabilities elicited — and frontier models already reason about it.
-
7.1%: What Happens When You Actually Measure Multi-Agent Safety
TrinityGuard tested real multi-agent system configurations against a structured, OWASP-grounded taxonomy of 20 risk types. The average safety pass rate was 7.1%. Here's what that number means and what the framework gives you to act on it.
-
Poisoning What Your Agent Remembers: The Cross-Session Attack You Haven't Modeled
eTAMP shows that a single compromised webpage can silently corrupt an agent's persistent memory, then trigger the payload on a completely different site in a future session — with attack success rates climbing to 32.5% when the agent is under stress.
-
No Auth Required: How a Healthcare RAG Chatbot Leaked 1,000 Patient Conversations
Researchers used nothing but Chrome DevTools to extract the system prompt, full RAG configuration, knowledge base, and 1,000 stored patient conversations from a live medical chatbot. The exploit wasn't prompt injection — it was basic web application security failure.
-
When AI Agents Talk in Embeddings, Text-Level Safety Filters Go Blind
RecursiveMAS replaces inter-agent text communication with latent-space embeddings for efficiency. The security consequence: an entirely new attack surface — latent-space injection — where adversarial representations propagate between agents with no text transcript, no content filter, and no audit trail.
-
Safe Agents, Unsafe Systems: The Non-Compositionality Problem in Multi-Agent Security
A 24-author paper from Oxford, CMU, MIT, and the Turing Institute argues that individually safe AI agents can compose into unsafe systems — and that securing each agent in isolation misses the point entirely.
-
What Red-Teaming Misses When Agents Talk to Each Other
Microsoft Research red-teamed a live 100+ agent platform and found four attack classes — worms, amplification, trust capture, proxy chains — that only emerge at network scale. Single-agent benchmarks miss all of them.
-
Your Guardrails Can't Read JSON: The Structural Bottleneck in Agentic Safety
New research finds that guardrail performance on tool-call trajectories correlates at ρ=0.79 with structured-data reasoning ability — and near-zero with jailbreak robustness. Here's what that means for how you secure agents.
-
Your Agent Is Mine: The LLM Router Supply Chain Attack You're Not Defending Against
Researchers bought 428 LLM API routers and found 9 actively injecting malicious code. Here's what that means for every agent that uses a third-party API proxy.
-
Three Papers, Three Attack Layers: Agent Security Gets Mapped
In one week, three independent research groups dissected the conversation, tool-use, and capability layers of AI agent systems. Here's what practitioners need to know.