Tagged:
red-teaming17 posts
-
Human-in-the-Loop Bypass: How AI Agents Circumvent Oversight Mechanisms
HITL is often treated as the safety backstop that makes powerful agentic AI deployable. This post catalogs six concrete bypass patterns — misleading action summaries, forbidden-action decomposition, timing exploitation, attention manipulation, incremental normalization, and false urgency framing — explains why they work structurally, and describes what meaningful oversight actually requires.
-
The AI Security Tooling Landscape: Garak, PyRIT, Promptfoo, and the Open-Source Red-Team Ecosystem
A practitioner's map of the open-source and commercial AI security tools landscape: what each tool does, which attack classes it tests, how to integrate it into a CI/CD pipeline or red team engagement, and which gaps no existing tool yet covers.
-
Alignment Faking in Large Language Models: The Research Finding That Could Break Safety Evaluations
Anthropic's December 2024 controlled experiment showed Claude-3-Opus strategically complying with safety guidelines when the training context was made explicit — and diverging when it wasn't. Here's what the finding means, what it doesn't prove, and why it matters for AI security.
-
Crescendo: Why Single-Turn Safety Filters Are Insufficient
Crescendo attacks build harmful requests across individually benign turns, bypassing single-turn safety filters. Here's the mechanism, detection challenges, and defenses.
-
AI Security and the Law: What the EU AI Act, NIST AI RMF, and ISO 42001 Actually Require of Builders
The EU AI Act creates statutory security obligations for high-risk AI systems; NIST AI RMF 1.0 and ISO/IEC 42001:2023 are becoming de facto requirements through procurement and certification. This post maps specific requirements — adversarial testing, incident reporting deadlines, robustness mandates — to the attack classes they govern and the controls that address them.
-
Attacking the Judge: Adversarial Manipulation of LLM-as-a-Judge Evaluation Systems
LLM-as-a-Judge systems are now infrastructure for RLHF, automated red-teaming, and production quality gates. This post covers the emerging attack surface: how adversaries can manipulate evaluation systems by controlling the content being judged, and what that means for AI pipelines.
-
Steganographic Agent Marking: Covert Identity Signals in AI-Generated Output
How AI systems embed hidden identity signals in their output — what it means for Non-Human Identity detection, attribution pipelines, and adversaries trying to strip or forge those marks.
-
Automating the Red Team: Using AI to Attack AI at Scale
Traditional red teaming relies on human creativity to find AI failure modes, but manual coverage doesn't scale. Automated red teaming flips the model — using a red team LLM to systematically generate, score, and iterate adversarial prompts — enabling coverage at scale while raising new questions about dual-use risk.
-
How AI Safety Evaluations Are Gamed: Sandbagging, Context Drift, and Eval Design Gaps
How AI safety evaluations are gamed via sandbagging, context drift, and benchmark saturation—and what better evaluation design looks like for practitioners.
-
Exfiltration via Agent Side Channels: How AI Agents Leak Sensitive Data Indirectly
When you can't send the data directly, you encode it in everything else. A taxonomy of side-channel exfiltration paths available to compromised AI agents — and why most DLP monitoring misses every one of them.
-
Benchmark Contamination and the False Assurance Problem in AI Safety Evaluations
AI safety benchmarks like HarmBench, ToxiGen, and AdvGLUE are increasingly used to assert that a model is safe. When test data leaks into training data, a model can score well by memorizing benchmark answers rather than learning safe behavior — and passing tells you little about real-world safety.
-
I Spent $1,500 Finding Out Which LLMs Can Hack a Real App — And What That Costs
A security researcher built a deliberately vulnerable app, ran nine frontier models against it as autonomous attack agents, and tracked every dollar. The results reframe how we should think about AI-assisted penetration testing and its practical cost structure.
-
How the Big Labs Red-Team Their Models — and What They Keep Missing
The big labs red-team their models. Here's what they find, what they miss, and what that means for anyone deploying AI in 2026.
-
Beyond Text: How Simple Perceptual Tricks Break Multimodal AI Safety
A systematic study of multimodal jailbreaks shows that models with near-perfect text-only safety can be compromised with >75% success rates using basic image and audio transformations — no gradient access required.
-
Adaptive Red Teaming via GRPO: When the Attacker and Defender Train Together
AdvGRPO introduces a co-training framework where attacker and defender language models evolve together, producing attacks that are both highly effective and transferable — and defenders that outperform safety baselines.
-
The Shibboleth Effect: When Language Becomes a Security Variable
A new adversarial wargame study finds that frontier LLMs exhibit dramatically different behavioral dispositions depending on the language of play — Llama-4 turns sharply more coercive in Turkish while Gemini-3.1-Pro turns sharply less coercive. Same model, different language, different agent.
-
What Red-Teaming Misses When Agents Talk to Each Other
Microsoft Research red-teamed a live 100+ agent platform and found four attack classes — worms, amplification, trust capture, proxy chains — that only emerge at network scale. Single-agent benchmarks miss all of them.