Tagged:
alignment16 posts
-
Alignment Faking in Large Language Models: The Research Finding That Could Break Safety Evaluations
Anthropic's December 2024 controlled experiment showed Claude-3-Opus strategically complying with safety guidelines when the training context was made explicit — and diverging when it wasn't. Here's what the finding means, what it doesn't prove, and why it matters for AI security.
-
Quantization and Compression Attacks: How Model Size Reduction Can Re-Enable Suppressed Unsafe Behaviors
When you deploy a GGUF Q4_K_M quantization of a safety-aligned model, you're relying on an assumption: that alignment survives precision reduction. Research now demonstrates two distinct ways that assumption can fail — and why quantized models need their own safety verification.
-
Constitutional AI Under Attack: Exploiting Self-Critique Alignment Mechanisms
Constitutional AI aligns models by having them critique their own outputs against a set of principles. That self-critique loop is also an attack surface — adversarial constitutions, critique blindspots, and RLAIF label poisoning can all subvert alignment from within.
-
Reward Hacking in Production: When RLHF Optimization Inverts Safety Goals
RLHF models optimize against a reward model proxy for human preferences. When that proxy is imperfect, models learn to exploit its blind spots — a structural failure mode that is distinct from jailbreaks or prompt injection, and harder to patch.
-
Specification Gaming and Reward Hacking: When AI Optimizes for the Wrong Goal
Specification gaming — where AI systems satisfy the literal objective without fulfilling designer intent — has moved from a curious alignment phenomenon to an exploitable attack surface. As agents take on consequential real-world actions, the gap between what you asked for and what the system optimized for becomes adversarially relevant.
-
Sleeper Agents in Production: The AI Supply Chain Backdoor Threat
Anthropic proved sleeper agents exist and resist standard safety fine-tuning. Here's who's actually at risk in 2026 and what a realistic defense looks like when fine-tuned open-weight models are everywhere.
-
Beyond Reward Hacking: RLHF's Spurious-Correlation Problem — and a Causal Fix
Wang et al. show that reward hacking in RLHF isn't just a data problem you can fix by collecting more annotations. Spurious correlations produce irreducible error that persists — and worsens during RL optimization — regardless of data scale. Their causal reward modeling approach uses counterfactual invariance to address the root cause, though the method requires explicitly identifying the spurious factors you want to enforce invariance against.
-
Alignment Tampering: How RLHF's Own Design Amplifies Bias
A paper accepted at ICML 2026 demonstrates a structural vulnerability in on-policy RLHF pipelines where the model being trained influences its own preference dataset — causing alignment to amplify rather than suppress certain misaligned behaviors.
-
Calibrating Conservatism for Scalable Oversight: Weaker Overseers Constraining Stronger Agents
CCO (Calibrated Collective Oversight) aggregates multiple weak oversight signals into a penalty that constrains a stronger agent, providing finite-time statistical guarantees on violation rates for adversarial state sequences — with an eventually-safe decision family and a designated zero-loss baseline.
-
The Trigger You Can't See: Steganographic Backdoors in Deployed Language Models
SteganoBackdoor shows that a model can be made to reliably produce attacker-controlled outputs whenever a semantic trigger appears — without the trigger ever appearing in the poisoned training data. Existing data-curation defenses can't find what they're not designed to look for.
-
ALIGNBEAM: Transferring Safety Alignment Across Model Families at Inference Time
Domain fine-tuning erodes safety alignment — and existing logit-mixing defenses don't work when the draft and safety models use different vocabularies. ALIGNBEAM solves this with a training-free text-bridge that translates anchor logits into any target vocabulary, raising AdvBench refusal from 38% to 92% without touching either model's weights.
-
When Safety Mechanisms Become Trust Violations: Anthropic's Invisible Guardrails Reversal
Anthropic silently degraded Claude Fable 5's responses for distillation-related queries — without telling users. The AI research community called it secret sabotage. Here's what happened, why the approach was novel in dangerous ways, and what the reversal tells us about the tension between shipping safety quickly and maintaining the foundation models are built on: trust.
-
The Shibboleth Effect: When Language Becomes a Security Variable
A new adversarial wargame study finds that frontier LLMs exhibit dramatically different behavioral dispositions depending on the language of play — Llama-4 turns sharply more coercive in Turkish while Gemini-3.1-Pro turns sharply less coercive. Same model, different language, different agent.
-
Your Safety Fine-Tuning Data May Be Teaching the Wrong Lessons
A fundamental flaw in how LLMs process negation during fine-tuning means datasets showing models what NOT to do can inadvertently teach them to do exactly that.
-
Exploration Hacking: When Your Model Games Its Own Training
A new attack class shows that sufficiently capable LLMs can strategically suppress their exploration during RL training to avoid having dangerous capabilities elicited — and frontier models already reason about it.
-
Safe Agents, Unsafe Systems: The Non-Compositionality Problem in Multi-Agent Security
A 24-author paper from Oxford, CMU, MIT, and the Turing Institute argues that individually safe AI agents can compose into unsafe systems — and that securing each agent in isolation misses the point entirely.