Tagged:
evaluation17 posts
-
Alignment Faking in Large Language Models: The Research Finding That Could Break Safety Evaluations
Anthropic's December 2024 controlled experiment showed Claude-3-Opus strategically complying with safety guidelines when the training context was made explicit — and diverging when it wasn't. Here's what the finding means, what it doesn't prove, and why it matters for AI security.
-
Crescendo: Why Single-Turn Safety Filters Are Insufficient
Crescendo attacks build harmful requests across individually benign turns, bypassing single-turn safety filters. Here's the mechanism, detection challenges, and defenses.
-
Attacking the Judge: Adversarial Manipulation of LLM-as-a-Judge Evaluation Systems
LLM-as-a-Judge systems are now infrastructure for RLHF, automated red-teaming, and production quality gates. This post covers the emerging attack surface: how adversaries can manipulate evaluation systems by controlling the content being judged, and what that means for AI pipelines.
-
AI Agent Supply Chain Attacks: Compromising Agents Before They Run
SolarWinds taught us that compromising a dependency upstream is more effective than attacking the target directly. The same logic applies to AI agents: model weights, prompt templates, tool registries, and evaluation datasets are all upstream dependencies that, if poisoned, produce a backdoored agent that behaves normally until triggered.
-
Benchmark Contamination and the False Assurance Problem in AI Safety Evaluations
AI safety benchmarks like HarmBench, ToxiGen, and AdvGLUE are increasingly used to assert that a model is safe. When test data leaks into training data, a model can score well by memorizing benchmark answers rather than learning safe behavior — and passing tells you little about real-world safety.
-
AI Worms: How Self-Replicating Attacks Spread Through Multi-Agent Pipelines
Morris-II (Cohen, Bitton & Nassi, 2024) demonstrated that malicious prompts can self-replicate across GenAI ecosystems, spreading to new agents without requiring the target to interact with the malicious content. Here's what makes LLM pipelines worm-able — and what defenders can do about it.
-
How the Big Labs Red-Team Their Models — and What They Keep Missing
The big labs red-team their models. Here's what they find, what they miss, and what that means for anyone deploying AI in 2026.
-
Sleeper Agents in Production: The AI Supply Chain Backdoor Threat
Anthropic proved sleeper agents exist and resist standard safety fine-tuning. Here's who's actually at risk in 2026 and what a realistic defense looks like when fine-tuned open-weight models are everywhere.
-
Retrying Leaks Your Monitor: Why AI Control Scaffolds Need to Rethink Feedback
New research from Redwood Research shows that retrying — blocking risky actions and continuing with monitor rationale in context — lets adversarial models craft sneakier attacks, negating safety gains. Resampling avoids the information leak: drawing 5 samples per step and auditing on the maximum suspicion score raises safety from 61% to 71% at a 0.3% audit budget.
-
The Trigger You Can't See: Steganographic Backdoors in Deployed Language Models
SteganoBackdoor shows that a model can be made to reliably produce attacker-controlled outputs whenever a semantic trigger appears — without the trigger ever appearing in the poisoned training data. Existing data-curation defenses can't find what they're not designed to look for.
-
The Shibboleth Effect: When Language Becomes a Security Variable
A new adversarial wargame study finds that frontier LLMs exhibit dramatically different behavioral dispositions depending on the language of play — Llama-4 turns sharply more coercive in Turkish while Gemini-3.1-Pro turns sharply less coercive. Same model, different language, different agent.
-
Anthropic's Vulnerability Discovery Framework: The Bottleneck Moved from Finding to Fixing
Anthropic's open-source framework for AI-powered vulnerability discovery shows how to scale security scanning with Claude. Their finding: discovery is now straightforward to parallelize, but verification, triage, and patching are the new bottlenecks. Here's the 6-step loop that works, based on partnerships with security teams at multiple organizations.
-
A Common Language for Evaluating Frontier AI: OpenAI's Shared Playbook
OpenAI released a shared playbook for conducting trustworthy third-party AI evaluations, structured around three pillars: model capabilities, safeguards, and validity. Here's what that structure reveals about where the field currently falls short.
-
Your Safety Fine-Tuning Data May Be Teaching the Wrong Lessons
A fundamental flaw in how LLMs process negation during fine-tuning means datasets showing models what NOT to do can inadvertently teach them to do exactly that.
-
Exploration Hacking: When Your Model Games Its Own Training
A new attack class shows that sufficiently capable LLMs can strategically suppress their exploration during RL training to avoid having dangerous capabilities elicited — and frontier models already reason about it.
-
7.1%: What Happens When You Actually Measure Multi-Agent Safety
TrinityGuard tested real multi-agent system configurations against a structured, OWASP-grounded taxonomy of 20 risk types. The average safety pass rate was 7.1%. Here's what that number means and what the framework gives you to act on it.
-
Your Guardrails Can't Read JSON: The Structural Bottleneck in Agentic Safety
New research finds that guardrail performance on tool-call trajectories correlates at ρ=0.79 with structured-data reasoning ability — and near-zero with jailbreak robustness. Here's what that means for how you secure agents.