Tagged:
llm-security30 posts
-
Cross-Tenant Data Leakage in Multi-Tenant LLM Deployments: Incidents, Architecture, and What to Demand from Providers
When an LLM service hosts multiple customers on shared infrastructure, failures in conversation isolation, prompt cache sharing, or conversation history storage can expose one customer's data to another. This post covers the documented failure modes, the March 2023 ChatGPT Redis incident, batch inference and system prompt bleed risks, and what enterprise architects should demand from LLM providers before trusting them with sensitive data.
-
Prompt Cache Timing Attacks: Side-Channel Leakage in LLM API Infrastructure
Major LLM providers cache prompt prefixes to reduce latency and cost — but shared cache infrastructure creates a timing side-channel. An attacker with access to a shared API key or multi-tenant deployment can measure whether a specific prefix is cached, leaking information about co-tenant system prompts and conversation context within the same cache namespace.
-
Slopsquatting: When AI Hallucinated Package Names Become a Supply Chain Attack
LLMs hallucinate package names at rates exceeding 20%. Attackers register those hallucinated names on npm and PyPI with malicious payloads. No typo required — the developer trusts the AI. This post explains the mechanics, documents the evidence, and gives developers concrete defenses.
-
WormGPT, FraudGPT, and the Criminal AI Ecosystem: Jailbroken Models as Cybercrime Infrastructure
A parallel AI ecosystem thrives on darknet forums and Telegram — jailbroken or purposely uncensored LLMs sold as Malware-as-a-Service that lower the barrier to phishing, BEC attacks, polymorphic malware, and automated fraud. What security teams need to know.
-
The AI Security Tooling Landscape: Garak, PyRIT, Promptfoo, and the Open-Source Red-Team Ecosystem
A practitioner's map of the open-source and commercial AI security tools landscape: what each tool does, which attack classes it tests, how to integrate it into a CI/CD pipeline or red team engagement, and which gaps no existing tool yet covers.
-
Crescendo: Why Single-Turn Safety Filters Are Insufficient
Crescendo attacks build harmful requests across individually benign turns, bypassing single-turn safety filters. Here's the mechanism, detection challenges, and defenses.
-
AI-Enabled Influence Operations: How LLMs Changed the Economics of Disinformation at Scale
LLMs don't just make disinformation faster — they fundamentally change the cost structure of influence operations, enabling persona networks, targeted narrative adaptation, and synthetic content generation at nation-state scale with small-team resources.
-
Securing the AI Inference Stack: GPU Memory Isolation, Model Serving Hardening, and Self-Hosted LLM Infrastructure Security
Self-hosted LLMs introduce attack surfaces below the application layer: GPU VRAM residuals between tenants, unauthenticated serving APIs, and unverified model weights on disk. This post maps the infrastructure threat model and provides a hardening checklist.
-
Defending Against Prompt Injection: Privilege Separation, Structured Outputs, and the Limits of Current Defenses
No single control stops prompt injection — but layering four concrete patterns (privilege separation, structured outputs, instruction hierarchy, spotlighting untrusted content) gives defenders a workable stack. This post synthesizes what actually works, and is honest about what remains unsolved.
-
Reasoning Model Security: Attacks on Chain-of-Thought and Extended Thinking
Reasoning models (o3, o4-mini, DeepSeek-R1, Claude with extended thinking) expose a distinct attack surface that standard LLM defenses don't fully address — attackers can manipulate the reasoning process itself, not just the inputs or outputs.
-
Training Data Extraction: How Attackers Query LLMs to Surface Memorized Private Content
LLMs verbatim-memorize chunks of their training data, and a simple prefix-completion attack can surface phone numbers, email addresses, code, and cryptographic identifiers that appeared in the training corpus — no model internals required. This post covers the mechanics, landmark empirical results, and the practical defenses that actually reduce extraction risk.
-
Adversarial Examples: The Foundational ML Attack That Still Breaks AI Systems in Production
Imperceptible perturbations that flip neural network classifications — from FGSM and PGD to physical-world stop-sign attacks and LLM adversarial suffixes. What adversarial examples are, why gradient-based attacks work, how defenses hold up, and what this means for production AI systems today.
-
LLM Guardrails in Practice: A Decision Guide to Runtime Input/Output Filtering Tools
LlamaGuard, Azure Prompt Shield, PromptGuard, NeMo Guardrails, Guardrails AI, and Presidio — how each works and when to deploy it.
-
AI as a Weapon: How Attackers Use LLMs Against Traditional Infrastructure
How LLM agents accelerate vulnerability discovery, exploit development, and reconnaissance against traditional software and infrastructure — and what this means for defenders.
-
Differential Privacy in Practice: What the Math Guarantees (and What It Doesn't) for AI Training Data
Differential privacy is the strongest mathematical privacy guarantee for ML training data — but the gap between formal ε-DP and deployed reality means many 'private' AI systems aren't nearly as private as advertised. This post bridges the math and the practice.
-
System Prompt Extraction: How Attackers Steal Proprietary AI Instructions
System prompts encode valuable business logic, persona, and safety overrides — and deployed AI products routinely leak them to motivated users. This post covers direct elicitation, behavioral inference, fine-tuning extraction, and the real-world incidents that showed extraction is not theoretical.
-
Hallucination as a Security Surface: Package Fabrication, Fake Credentials, and Confident Wrong Advice
LLM hallucination isn't just a reliability problem — it's an attack surface. Fabricated package names, fake credentials, and wrong security advice create real, exploitable gaps.
-
Side-Channel Attacks on LLM APIs: What Response Timing and Token Counts Reveal
How LLM API metadata — token counts, timing, and streaming patterns — leaks system prompt structure and model configuration to adversaries.
-
Cross-Tenant Contamination in LLM APIs: When Other Users' Context Leaks Into Your Session
Multi-tenant LLM deployments — shared inference infrastructure, KV cache reuse, batched requests — create subtle cross-tenant data exposure risks that differ from classical API security vulnerabilities. This post maps the threat surface unique to shared AI inference.
-
Prompt Injection in Long-Context Windows: When More Context Means More Attack Surface
Extended context windows (128K–1M tokens) introduce an underappreciated prompt injection attack surface: injected instructions buried deep in retrieved documents, conversation histories, or tool outputs can override system prompts or hijack reasoning far from the visible top of the context. This post maps the threat, explains why attention geography matters for defenders, and offers a practical mitigation checklist.
-
Automating the Red Team: Using AI to Attack AI at Scale
Traditional red teaming relies on human creativity to find AI failure modes, but manual coverage doesn't scale. Automated red teaming flips the model — using a red team LLM to systematically generate, score, and iterate adversarial prompts — enabling coverage at scale while raising new questions about dual-use risk.
-
Model Extraction via API Queries: Stealing Proprietary AI Without the Weights
Systematic API queries can reconstruct a proprietary model's behavior — and sometimes its architecture — without ever touching the weights. The security implications are dual: IP theft and a cheap path to an unconstrained surrogate for downstream adversarial attack crafting.
-
Shadow Prompting: How Hidden System Instructions Hijack AI Behavior
Hidden system-level instructions can silently override how an AI application behaves — without the user ever seeing the manipulation. This post explores what makes shadow prompting possible, how it differs from ordinary prompt injection, and what architectural choices actually reduce the risk.
-
Token Smuggling: Unicode Tricks That Slip Past AI Safety Filters
How attackers use Unicode homoglyphs, invisible characters, bidirectional overrides, and encoding tricks to smuggle disallowed content past LLM safety systems — and why the tokenizer-renderer gap is a structural attack surface that normalization alone doesn't fully close.
-
Adversarial Prompt Caching: Timing Attacks and Injection via Shared KV Caches
As LLM providers roll out shared prefix caching to cut inference costs, a new attack surface emerges: cache-timing side channels and cross-tenant injection vectors at the inference infrastructure layer.
-
Prompt Injection as Role Confusion: Why LLMs Trust Style Over Role Tags
New research reveals LLMs identify roles from writing style rather than structural tags — making the system role tag a suggestion, not a security boundary. Destyling adversarial text drops attack success from 61% to 10%. Here's what that means for defenders.
-
Jailbreak-as-a-Service: The Underground Market for LLM Exploit Techniques
Mapping the underground economy around LLM exploitation — paid jailbreak APIs, model-specific bypass markets, and the professionalization of AI adversaries.
-
Indirect Prompt Injection Against Production Systems: A Survey of Documented Disclosures
Five researcher-disclosed proof-of-concept exploits against production AI systems (2023–2024) reveal recurring structural patterns that defenders can act on today: from Bing Chat web-content sideloading to Microsoft 365 Copilot email exfiltration chains.
-
Your Spreadsheet Is the Attack Surface: ChatGPT for Google Sheets Data Exfiltration
A single hidden prompt in an imported sheet can silently exfiltrate your entire Google Drive workbook collection, replace the ChatGPT sidebar with an attacker-controlled phishing interface, and bypass the extension's own human-approval safety setting. PromptArmor's analysis of the ChatGPT for Google Sheets add-on shows how LLM extensions inherit every permission you've granted — and hand them to whoever controls the data you import.
-
One Prompt, One High-Profile Instagram Account: How Meta's Support Bot Became an Exploit
Hackers bypassed Meta's AI support chatbot with a single text prompt to take over high-profile Instagram accounts — including the Barack Obama White House account. The bot had account recovery powers, no human escalation path, and failed at its most critical function: distinguishing between the account owner and an attacker who knew the right words.