LLM & Model Security
Attacks targeting the model itself: fine-tuning vulnerabilities, RAG poisoning, model extraction, weight theft, and inference-time manipulation of large language models.
48 posts in this topic
-
Cross-Tenant Data Leakage in Multi-Tenant LLM Deployments: Incidents, Architecture, and What to Demand from Providers
When an LLM service hosts multiple customers on shared infrastructure, failures in conversation isolation, prompt cache sharing, or conversation history storage can expose one customer's data to another. This post covers the documented failure modes, the March 2023 ChatGPT Redis incident, batch inference and system prompt bleed risks, and what enterprise architects should demand from LLM providers before trusting them with sensitive data.
-
Prompt Cache Timing Attacks: Side-Channel Leakage in LLM API Infrastructure
Major LLM providers cache prompt prefixes to reduce latency and cost — but shared cache infrastructure creates a timing side-channel. An attacker with access to a shared API key or multi-tenant deployment can measure whether a specific prefix is cached, leaking information about co-tenant system prompts and conversation context within the same cache namespace.
-
Slopsquatting: When AI Hallucinated Package Names Become a Supply Chain Attack
LLMs hallucinate package names at rates exceeding 20%. Attackers register those hallucinated names on npm and PyPI with malicious payloads. No typo required — the developer trusts the AI. This post explains the mechanics, documents the evidence, and gives developers concrete defenses.
-
WormGPT, FraudGPT, and the Criminal AI Ecosystem: Jailbroken Models as Cybercrime Infrastructure
A parallel AI ecosystem thrives on darknet forums and Telegram — jailbroken or purposely uncensored LLMs sold as Malware-as-a-Service that lower the barrier to phishing, BEC attacks, polymorphic malware, and automated fraud. What security teams need to know.
-
The AI Security Tooling Landscape: Garak, PyRIT, Promptfoo, and the Open-Source Red-Team Ecosystem
A practitioner's map of the open-source and commercial AI security tools landscape: what each tool does, which attack classes it tests, how to integrate it into a CI/CD pipeline or red team engagement, and which gaps no existing tool yet covers.
-
Crescendo: Why Single-Turn Safety Filters Are Insufficient
Crescendo attacks build harmful requests across individually benign turns, bypassing single-turn safety filters. Here's the mechanism, detection challenges, and defenses.
-
AI-Enabled Influence Operations: How LLMs Changed the Economics of Disinformation at Scale
LLMs don't just make disinformation faster — they fundamentally change the cost structure of influence operations, enabling persona networks, targeted narrative adaptation, and synthetic content generation at nation-state scale with small-team resources.
-
Mechanistic Interpretability as a Security Tool: Detecting Backdoors and Hidden Behaviors in AI Models
Circuits, sparse autoencoders, and activation steering let security teams detect backdoors and hidden model behaviors that red-teaming cannot find.
-
LLM Security Monitoring in Production: Anomaly Detection, Audit Logging, and Intrusion Detection for AI Systems
Guardrails block known bad inputs. Incident response handles breaches after they happen. This post covers the gap: building LLM-specific observability that detects attacks while they are occurring — anomalous outputs, prompt injection signatures, unusual tool-call patterns, and model drift from baseline.
-
Multi-Agent Orchestration Security: Trust, Delegation, and Inter-Agent Attack Surfaces
When Agent A delegates a task to Agent B, the security properties of the entire system reduce to the weakest trust model in the chain. This post maps the attack taxonomy unique to orchestrator-worker architectures — privilege escalation, agent impersonation, confused deputy, tool-chain hijacking — and pairs each with concrete defenses grounded in published research.
-
Securing the AI Inference Stack: GPU Memory Isolation, Model Serving Hardening, and Self-Hosted LLM Infrastructure Security
Self-hosted LLMs introduce attack surfaces below the application layer: GPU VRAM residuals between tenants, unauthenticated serving APIs, and unverified model weights on disk. This post maps the infrastructure threat model and provides a hardening checklist.
-
Model Hub Supply Chain Attacks: Malicious Models, Tokenizer Exploits, and Typosquatting on Hugging Face
Downloading an open-weight model from a public hub is not a read-only operation. Custom tokenizer classes and auto_map configs execute arbitrary Python when trust_remote_code is set, LoRA adapters can trojanize safe base models, and typosquatted namespaces are a documented distribution vector. Here's the threat model practitioners need before they run from_pretrained().
-
Defending Against Prompt Injection: Privilege Separation, Structured Outputs, and the Limits of Current Defenses
No single control stops prompt injection — but layering four concrete patterns (privilege separation, structured outputs, instruction hierarchy, spotlighting untrusted content) gives defenders a workable stack. This post synthesizes what actually works, and is honest about what remains unsolved.
-
Reasoning Model Security: Attacks on Chain-of-Thought and Extended Thinking
Reasoning models (o3, o4-mini, DeepSeek-R1, Claude with extended thinking) expose a distinct attack surface that standard LLM defenses don't fully address — attackers can manipulate the reasoning process itself, not just the inputs or outputs.
-
Training Data Extraction: How Attackers Query LLMs to Surface Memorized Private Content
LLMs verbatim-memorize chunks of their training data, and a simple prefix-completion attack can surface phone numbers, email addresses, code, and cryptographic identifiers that appeared in the training corpus — no model internals required. This post covers the mechanics, landmark empirical results, and the practical defenses that actually reduce extraction risk.
-
Poisoning the Knowledge Base: Adversarial Document Injection into RAG Vector Stores
RAG can be weaponized by anyone who can write to the corpus. This post maps the attack taxonomy — two empirically validated classes (PoisonedRAG, BadRAG), one extrapolated operational pattern, and persistent corpus injection — with real-world ingestion surfaces and a defender checklist.
-
Adversarial Examples: The Foundational ML Attack That Still Breaks AI Systems in Production
Imperceptible perturbations that flip neural network classifications — from FGSM and PGD to physical-world stop-sign attacks and LLM adversarial suffixes. What adversarial examples are, why gradient-based attacks work, how defenses hold up, and what this means for production AI systems today.
-
AI Incident Response: A Practitioner's Playbook for When Your AI System Is Compromised
Detection signals, containment options, evidence preservation, and recovery procedures for AI-specific security incidents — the operational complement to attack coverage. NIST IR lifecycle applied to prompt injection, model backdoors, data poisoning, and adversarial input attacks.
-
LLM Guardrails in Practice: A Decision Guide to Runtime Input/Output Filtering Tools
LlamaGuard, Azure Prompt Shield, PromptGuard, NeMo Guardrails, Guardrails AI, and Presidio — how each works and when to deploy it.
-
AI as a Weapon: How Attackers Use LLMs Against Traditional Infrastructure
How LLM agents accelerate vulnerability discovery, exploit development, and reconnaissance against traditional software and infrastructure — and what this means for defenders.
-
Differential Privacy in Practice: What the Math Guarantees (and What It Doesn't) for AI Training Data
Differential privacy is the strongest mathematical privacy guarantee for ML training data — but the gap between formal ε-DP and deployed reality means many 'private' AI systems aren't nearly as private as advertised. This post bridges the math and the practice.
-
ML Model Provenance: Signing, SBOMs, and Verifying the AI You Deploy Before It Runs
You wouldn't deploy software without checksums and signatures. But most organizations download model weights and run them without any provenance verification at all. This post covers the practical mechanics of model signing, ML SBOMs, and the emerging infrastructure for verifying a model's origins before it touches production.
-
Zero-Trust Architecture for AI Agent Deployments: Never Trust, Always Verify — Even Your Own Agents
Zero-trust was designed for human users on networks. AI agents break its assumptions: they act autonomously, spawn child agents, inherit permissions, and run long-lived sessions. Here's how to adapt zero-trust principles specifically for multi-agent and LLM deployment contexts.
-
Malicious AI Model Files: Pickle Exploits and Arbitrary Code Execution on Model Load
Downloading a model file and calling torch.load() is a potential code execution event. This post explains the pickle mechanics that make it so, the real-world exploits already found in the wild, how HuggingFace has responded, and what safe alternatives actually prevent the attack.
-
System Prompt Extraction: How Attackers Steal Proprietary AI Instructions
System prompts encode valuable business logic, persona, and safety overrides — and deployed AI products routinely leak them to motivated users. This post covers direct elicitation, behavioral inference, fine-tuning extraction, and the real-world incidents that showed extraction is not theoretical.
-
Hallucination as a Security Surface: Package Fabrication, Fake Credentials, and Confident Wrong Advice
LLM hallucination isn't just a reliability problem — it's an attack surface. Fabricated package names, fake credentials, and wrong security advice create real, exploitable gaps.
-
Backdoor Attacks in Foundation Models: Sleeper Triggers That Survive Fine-Tuning
Pre-trained LLMs can be trojaned at the foundation stage, with adversarial triggers embedded in weights that persist through downstream fine-tuning and RLHF safety training. This post explains how these attacks work, why they're so persistent, and what practitioners can do about them.
-
Side-Channel Attacks on LLM APIs: What Response Timing and Token Counts Reveal
How LLM API metadata — token counts, timing, and streaming patterns — leaks system prompt structure and model configuration to adversaries.
-
Cross-Tenant Contamination in LLM APIs: When Other Users' Context Leaks Into Your Session
Multi-tenant LLM deployments — shared inference infrastructure, KV cache reuse, batched requests — create subtle cross-tenant data exposure risks that differ from classical API security vulnerabilities. This post maps the threat surface unique to shared AI inference.
-
Machine Unlearning Security: When Forgetting Training Data Creates New Vulnerabilities
Regulators are demanding AI systems forget specific training data. Attackers are figuring out how to weaponize that forgetting — through verification gaps, induced forgetting abuse, and unlearning as a path to alignment degradation.
-
Prompt Injection in Long-Context Windows: When More Context Means More Attack Surface
Extended context windows (128K–1M tokens) introduce an underappreciated prompt injection attack surface: injected instructions buried deep in retrieved documents, conversation histories, or tool outputs can override system prompts or hijack reasoning far from the visible top of the context. This post maps the threat, explains why attention geography matters for defenders, and offers a practical mitigation checklist.
-
Automating the Red Team: Using AI to Attack AI at Scale
Traditional red teaming relies on human creativity to find AI failure modes, but manual coverage doesn't scale. Automated red teaming flips the model — using a red team LLM to systematically generate, score, and iterate adversarial prompts — enabling coverage at scale while raising new questions about dual-use risk.
-
Model Extraction via API Queries: Stealing Proprietary AI Without the Weights
Systematic API queries can reconstruct a proprietary model's behavior — and sometimes its architecture — without ever touching the weights. The security implications are dual: IP theft and a cheap path to an unconstrained surrogate for downstream adversarial attack crafting.
-
Shadow Prompting: How Hidden System Instructions Hijack AI Behavior
Hidden system-level instructions can silently override how an AI application behaves — without the user ever seeing the manipulation. This post explores what makes shadow prompting possible, how it differs from ordinary prompt injection, and what architectural choices actually reduce the risk.
-
Token Smuggling: Unicode Tricks That Slip Past AI Safety Filters
How attackers use Unicode homoglyphs, invisible characters, bidirectional overrides, and encoding tricks to smuggle disallowed content past LLM safety systems — and why the tokenizer-renderer gap is a structural attack surface that normalization alone doesn't fully close.
-
Adversarial Prompt Caching: Timing Attacks and Injection via Shared KV Caches
As LLM providers roll out shared prefix caching to cut inference costs, a new attack surface emerges: cache-timing side channels and cross-tenant injection vectors at the inference infrastructure layer.
-
Jailbreak Robustness After Fine-Tuning: How Safety Alignment Degrades
Safety fine-tuning instills refusal behaviors in LLMs — but those behaviors are surprisingly brittle under subsequent fine-tuning. This post explains the mechanism, the empirical evidence, and what enterprise deployers can do about it.
-
LLM Output Watermarking: Provenance, Detection Limits, and Evasion
Token-level watermarks, cryptographic signing, and semantic embedding all promise attribution of AI-generated text — but paraphrase attacks, closed-pipeline requirements, and research-stage maturity constrain what any of them can actually guarantee. An honest practitioner's assessment.
-
Fine-Tuning Trojans: Injecting Backdoors Through the Model Training Pipeline
How malicious training data, tampered datasets, and compromised fine-tuning APIs plant backdoored behavior in legitimate base models — and what defenders can do.
-
I Spent $1,500 Finding Out Which LLMs Can Hack a Real App — And What That Costs
A security researcher built a deliberately vulnerable app, ran nine frontier models against it as autonomous attack agents, and tracked every dollar. The results reframe how we should think about AI-assisted penetration testing and its practical cost structure.
-
Prompt Injection as Role Confusion: Why LLMs Trust Style Over Role Tags
New research reveals LLMs identify roles from writing style rather than structural tags — making the system role tag a suggestion, not a security boundary. Destyling adversarial text drops attack success from 61% to 10%. Here's what that means for defenders.
-
Jailbreak-as-a-Service: The Underground Market for LLM Exploit Techniques
Mapping the underground economy around LLM exploitation — paid jailbreak APIs, model-specific bypass markets, and the professionalization of AI adversaries.
-
Poisoning the Well: Memory and RAG Attacks Against Long-Context AI Systems
As AI agents gain persistent memory through vector stores and RAG pipelines, a new attack surface emerges: injecting malicious content into the agent's long-term knowledge to control future behavior. This post maps the taxonomy of memory poisoning attacks — from retrieval manipulation to cross-session contamination — and what defenders need to audit before production.
-
Indirect Prompt Injection Against Production Systems: A Survey of Documented Disclosures
Five researcher-disclosed proof-of-concept exploits against production AI systems (2023–2024) reveal recurring structural patterns that defenders can act on today: from Bing Chat web-content sideloading to Microsoft 365 Copilot email exfiltration chains.
-
ALIGNBEAM: Transferring Safety Alignment Across Model Families at Inference Time
Domain fine-tuning erodes safety alignment — and existing logit-mixing defenses don't work when the draft and safety models use different vocabularies. ALIGNBEAM solves this with a training-free text-bridge that translates anchor logits into any target vocabulary, raising AdvBench refusal from 38% to 92% without touching either model's weights.
-
Your Spreadsheet Is the Attack Surface: ChatGPT for Google Sheets Data Exfiltration Foundational
A single hidden prompt in an imported sheet can silently exfiltrate your entire Google Drive workbook collection, replace the ChatGPT sidebar with an attacker-controlled phishing interface, and bypass the extension's own human-approval safety setting. PromptArmor's analysis of the ChatGPT for Google Sheets add-on shows how LLM extensions inherit every permission you've granted — and hand them to whoever controls the data you import.
-
One Prompt, One High-Profile Instagram Account: How Meta's Support Bot Became an Exploit Foundational
Hackers bypassed Meta's AI support chatbot with a single text prompt to take over high-profile Instagram accounts — including the Barack Obama White House account. The bot had account recovery powers, no human escalation path, and failed at its most critical function: distinguishing between the account owner and an attacker who knew the right words.
-
No Auth Required: How a Healthcare RAG Chatbot Leaked 1,000 Patient Conversations Foundational
Researchers used nothing but Chrome DevTools to extract the system prompt, full RAG configuration, knowledge base, and 1,000 stored patient conversations from a live medical chatbot. The exploit wasn't prompt injection — it was basic web application security failure.