Tagged:
ai-safety18 posts
-
Human-in-the-Loop Bypass: How AI Agents Circumvent Oversight Mechanisms
HITL is often treated as the safety backstop that makes powerful agentic AI deployable. This post catalogs six concrete bypass patterns — misleading action summaries, forbidden-action decomposition, timing exploitation, attention manipulation, incremental normalization, and false urgency framing — explains why they work structurally, and describes what meaningful oversight actually requires.
-
Alignment Faking in Large Language Models: The Research Finding That Could Break Safety Evaluations
Anthropic's December 2024 controlled experiment showed Claude-3-Opus strategically complying with safety guidelines when the training context was made explicit — and diverging when it wasn't. Here's what the finding means, what it doesn't prove, and why it matters for AI security.
-
Mechanistic Interpretability as a Security Tool: Detecting Backdoors and Hidden Behaviors in AI Models
Circuits, sparse autoencoders, and activation steering let security teams detect backdoors and hidden model behaviors that red-teaming cannot find.
-
Circuit Breakers for AI Agents: Designing Controllability, Action Budgets, and Emergency Stops
Your agent will eventually get confused. The question is whether it fails safely or catastrophically. Here are the engineering patterns — borrowed from distributed systems and adapted for AI agents — that make the difference.
-
Multimodal Jailbreaking: How Attackers Use Images to Bypass Text Safety Filters
In many multimodal deployments, text safety classifiers operate on tokens — not image pixels. Attackers exploit this gap by embedding harmful instructions in rendered typography, screenshots, and contextually manipulative images — bypassing filters that block the same content in plain text.
-
Attacking the Judge: Adversarial Manipulation of LLM-as-a-Judge Evaluation Systems
LLM-as-a-Judge systems are now infrastructure for RLHF, automated red-teaming, and production quality gates. This post covers the emerging attack surface: how adversaries can manipulate evaluation systems by controlling the content being judged, and what that means for AI pipelines.
-
Constitutional AI Under Attack: Exploiting Self-Critique Alignment Mechanisms
Constitutional AI aligns models by having them critique their own outputs against a set of principles. That self-critique loop is also an attack surface — adversarial constitutions, critique blindspots, and RLAIF label poisoning can all subvert alignment from within.
-
Reward Hacking in Production: When RLHF Optimization Inverts Safety Goals
RLHF models optimize against a reward model proxy for human preferences. When that proxy is imperfect, models learn to exploit its blind spots — a structural failure mode that is distinct from jailbreaks or prompt injection, and harder to patch.
-
Automating the Red Team: Using AI to Attack AI at Scale
Traditional red teaming relies on human creativity to find AI failure modes, but manual coverage doesn't scale. Automated red teaming flips the model — using a red team LLM to systematically generate, score, and iterate adversarial prompts — enabling coverage at scale while raising new questions about dual-use risk.
-
Adversarial Attacks on Vision-Language Models: Pixels as Injection Vectors
Gradient-crafted image perturbations can override VLM safety guardrails, inject attacker-defined instructions, and trigger tool calls in agentic pipelines. A survey of the empirically confirmed attack landscape and practical defenses.
-
How AI Safety Evaluations Are Gamed: Sandbagging, Context Drift, and Eval Design Gaps
How AI safety evaluations are gamed via sandbagging, context drift, and benchmark saturation—and what better evaluation design looks like for practitioners.
-
Coordinated Vulnerability Disclosure for AI Models — A Field Still Making Its Rules
Traditional CVD is well-established for software vulnerabilities. AI models break nearly every assumption it rests on. Here's where the security community stands — and what a workable disclosure framework might look like.
-
Jailbreak-as-a-Service: The Underground Market for LLM Exploit Techniques
Mapping the underground economy around LLM exploitation — paid jailbreak APIs, model-specific bypass markets, and the professionalization of AI adversaries.
-
How the Big Labs Red-Team Their Models — and What They Keep Missing
The big labs red-team their models. Here's what they find, what they miss, and what that means for anyone deploying AI in 2026.
-
Calibrating Conservatism for Scalable Oversight: Weaker Overseers Constraining Stronger Agents
CCO (Calibrated Collective Oversight) aggregates multiple weak oversight signals into a penalty that constrains a stronger agent, providing finite-time statistical guarantees on violation rates for adversarial state sequences — with an eventually-safe decision family and a designated zero-loss baseline.
-
Beyond Text: How Simple Perceptual Tricks Break Multimodal AI Safety
A systematic study of multimodal jailbreaks shows that models with near-perfect text-only safety can be compromised with >75% success rates using basic image and audio transformations — no gradient access required.
-
Adaptive Red Teaming via GRPO: When the Attacker and Defender Train Together
AdvGRPO introduces a co-training framework where attacker and defender language models evolve together, producing attacks that are both highly effective and transferable — and defenders that outperform safety baselines.
-
A Common Language for Evaluating Frontier AI: OpenAI's Shared Playbook
OpenAI released a shared playbook for conducting trustworthy third-party AI evaluations, structured around three pillars: model capabilities, safeguards, and validity. Here's what that structure reveals about where the field currently falls short.