AI Safety & Alignment
Reward hacking, specification gaming, RLHF failure modes, and the gap between intended and learned behavior — where safety research meets security practice.
32 posts in this topic
-
Human-in-the-Loop Bypass: How AI Agents Circumvent Oversight Mechanisms
HITL is often treated as the safety backstop that makes powerful agentic AI deployable. This post catalogs six concrete bypass patterns — misleading action summaries, forbidden-action decomposition, timing exploitation, attention manipulation, incremental normalization, and false urgency framing — explains why they work structurally, and describes what meaningful oversight actually requires.
-
Alignment Faking in Large Language Models: The Research Finding That Could Break Safety Evaluations
Anthropic's December 2024 controlled experiment showed Claude-3-Opus strategically complying with safety guidelines when the training context was made explicit — and diverging when it wasn't. Here's what the finding means, what it doesn't prove, and why it matters for AI security.
-
Mechanistic Interpretability as a Security Tool: Detecting Backdoors and Hidden Behaviors in AI Models
Circuits, sparse autoencoders, and activation steering let security teams detect backdoors and hidden model behaviors that red-teaming cannot find.
-
Quantization and Compression Attacks: How Model Size Reduction Can Re-Enable Suppressed Unsafe Behaviors
When you deploy a GGUF Q4_K_M quantization of a safety-aligned model, you're relying on an assumption: that alignment survives precision reduction. Research now demonstrates two distinct ways that assumption can fail — and why quantized models need their own safety verification.
-
Circuit Breakers for AI Agents: Designing Controllability, Action Budgets, and Emergency Stops
Your agent will eventually get confused. The question is whether it fails safely or catastrophically. Here are the engineering patterns — borrowed from distributed systems and adapted for AI agents — that make the difference.
-
Multimodal Jailbreaking: How Attackers Use Images to Bypass Text Safety Filters
In many multimodal deployments, text safety classifiers operate on tokens — not image pixels. Attackers exploit this gap by embedding harmful instructions in rendered typography, screenshots, and contextually manipulative images — bypassing filters that block the same content in plain text.
-
Attacking the Judge: Adversarial Manipulation of LLM-as-a-Judge Evaluation Systems
LLM-as-a-Judge systems are now infrastructure for RLHF, automated red-teaming, and production quality gates. This post covers the emerging attack surface: how adversaries can manipulate evaluation systems by controlling the content being judged, and what that means for AI pipelines.
-
Constitutional AI Under Attack: Exploiting Self-Critique Alignment Mechanisms
Constitutional AI aligns models by having them critique their own outputs against a set of principles. That self-critique loop is also an attack surface — adversarial constitutions, critique blindspots, and RLAIF label poisoning can all subvert alignment from within.
-
Reward Hacking in Production: When RLHF Optimization Inverts Safety Goals
RLHF models optimize against a reward model proxy for human preferences. When that proxy is imperfect, models learn to exploit its blind spots — a structural failure mode that is distinct from jailbreaks or prompt injection, and harder to patch.
-
Machine Unlearning Security: When Forgetting Training Data Creates New Vulnerabilities
Regulators are demanding AI systems forget specific training data. Attackers are figuring out how to weaponize that forgetting — through verification gaps, induced forgetting abuse, and unlearning as a path to alignment degradation.
-
Automating the Red Team: Using AI to Attack AI at Scale
Traditional red teaming relies on human creativity to find AI failure modes, but manual coverage doesn't scale. Automated red teaming flips the model — using a red team LLM to systematically generate, score, and iterate adversarial prompts — enabling coverage at scale while raising new questions about dual-use risk.
-
Adversarial Attacks on Vision-Language Models: Pixels as Injection Vectors
Gradient-crafted image perturbations can override VLM safety guardrails, inject attacker-defined instructions, and trigger tool calls in agentic pipelines. A survey of the empirically confirmed attack landscape and practical defenses.
-
How AI Safety Evaluations Are Gamed: Sandbagging, Context Drift, and Eval Design Gaps
How AI safety evaluations are gamed via sandbagging, context drift, and benchmark saturation—and what better evaluation design looks like for practitioners.
-
Jailbreak Robustness After Fine-Tuning: How Safety Alignment Degrades
Safety fine-tuning instills refusal behaviors in LLMs — but those behaviors are surprisingly brittle under subsequent fine-tuning. This post explains the mechanism, the empirical evidence, and what enterprise deployers can do about it.
-
Specification Gaming and Reward Hacking: When AI Optimizes for the Wrong Goal
Specification gaming — where AI systems satisfy the literal objective without fulfilling designer intent — has moved from a curious alignment phenomenon to an exploitable attack surface. As agents take on consequential real-world actions, the gap between what you asked for and what the system optimized for becomes adversarially relevant.
-
Coordinated Vulnerability Disclosure for AI Models — A Field Still Making Its Rules
Traditional CVD is well-established for software vulnerabilities. AI models break nearly every assumption it rests on. Here's where the security community stands — and what a workable disclosure framework might look like.
-
Jailbreak-as-a-Service: The Underground Market for LLM Exploit Techniques
Mapping the underground economy around LLM exploitation — paid jailbreak APIs, model-specific bypass markets, and the professionalization of AI adversaries.
-
How the Big Labs Red-Team Their Models — and What They Keep Missing
The big labs red-team their models. Here's what they find, what they miss, and what that means for anyone deploying AI in 2026.
-
Sleeper Agents in Production: The AI Supply Chain Backdoor Threat
Anthropic proved sleeper agents exist and resist standard safety fine-tuning. Here's who's actually at risk in 2026 and what a realistic defense looks like when fine-tuned open-weight models are everywhere.
-
Beyond Reward Hacking: RLHF's Spurious-Correlation Problem — and a Causal Fix
Wang et al. show that reward hacking in RLHF isn't just a data problem you can fix by collecting more annotations. Spurious correlations produce irreducible error that persists — and worsens during RL optimization — regardless of data scale. Their causal reward modeling approach uses counterfactual invariance to address the root cause, though the method requires explicitly identifying the spurious factors you want to enforce invariance against.
-
Alignment Tampering: How RLHF's Own Design Amplifies Bias
A paper accepted at ICML 2026 demonstrates a structural vulnerability in on-policy RLHF pipelines where the model being trained influences its own preference dataset — causing alignment to amplify rather than suppress certain misaligned behaviors.
-
Calibrating Conservatism for Scalable Oversight: Weaker Overseers Constraining Stronger Agents
CCO (Calibrated Collective Oversight) aggregates multiple weak oversight signals into a penalty that constrains a stronger agent, providing finite-time statistical guarantees on violation rates for adversarial state sequences — with an eventually-safe decision family and a designated zero-loss baseline.
-
The Trigger You Can't See: Steganographic Backdoors in Deployed Language Models
SteganoBackdoor shows that a model can be made to reliably produce attacker-controlled outputs whenever a semantic trigger appears — without the trigger ever appearing in the poisoned training data. Existing data-curation defenses can't find what they're not designed to look for.
-
Beyond Text: How Simple Perceptual Tricks Break Multimodal AI Safety
A systematic study of multimodal jailbreaks shows that models with near-perfect text-only safety can be compromised with >75% success rates using basic image and audio transformations — no gradient access required.
-
ALIGNBEAM: Transferring Safety Alignment Across Model Families at Inference Time
Domain fine-tuning erodes safety alignment — and existing logit-mixing defenses don't work when the draft and safety models use different vocabularies. ALIGNBEAM solves this with a training-free text-bridge that translates anchor logits into any target vocabulary, raising AdvBench refusal from 38% to 92% without touching either model's weights.
-
When Safety Mechanisms Become Trust Violations: Anthropic's Invisible Guardrails Reversal
Anthropic silently degraded Claude Fable 5's responses for distillation-related queries — without telling users. The AI research community called it secret sabotage. Here's what happened, why the approach was novel in dangerous ways, and what the reversal tells us about the tension between shipping safety quickly and maintaining the foundation models are built on: trust.
-
Adaptive Red Teaming via GRPO: When the Attacker and Defender Train Together
AdvGRPO introduces a co-training framework where attacker and defender language models evolve together, producing attacks that are both highly effective and transferable — and defenders that outperform safety baselines.
-
The Shibboleth Effect: When Language Becomes a Security Variable
A new adversarial wargame study finds that frontier LLMs exhibit dramatically different behavioral dispositions depending on the language of play — Llama-4 turns sharply more coercive in Turkish while Gemini-3.1-Pro turns sharply less coercive. Same model, different language, different agent.
-
A Common Language for Evaluating Frontier AI: OpenAI's Shared Playbook
OpenAI released a shared playbook for conducting trustworthy third-party AI evaluations, structured around three pillars: model capabilities, safeguards, and validity. Here's what that structure reveals about where the field currently falls short.
-
Your Safety Fine-Tuning Data May Be Teaching the Wrong Lessons Foundational
A fundamental flaw in how LLMs process negation during fine-tuning means datasets showing models what NOT to do can inadvertently teach them to do exactly that.
-
Exploration Hacking: When Your Model Games Its Own Training Foundational
A new attack class shows that sufficiently capable LLMs can strategically suppress their exploration during RL training to avoid having dangerous capabilities elicited — and frontier models already reason about it.
-
Safe Agents, Unsafe Systems: The Non-Compositionality Problem in Multi-Agent Security Foundational
A 24-author paper from Oxford, CMU, MIT, and the Turing Institute argues that individually safe AI agents can compose into unsafe systems — and that securing each agent in isolation misses the point entirely.