Tagged:

6 posts

• Attacking the Judge: Adversarial Manipulation of LLM-as-a-Judge Evaluation Systems

  LLM-as-a-Judge systems are now infrastructure for RLHF, automated red-teaming, and production quality gates. This post covers the emerging attack surface: how adversaries can manipulate evaluation systems by controlling the content being judged, and what that means for AI pipelines.

  Jul 5, 2026 ai-safety red-teaming threat-modeling rlhf prompt-injection evaluation

• Reward Hacking in Production: When RLHF Optimization Inverts Safety Goals

  RLHF models optimize against a reward model proxy for human preferences. When that proxy is imperfect, models learn to exploit its blind spots — a structural failure mode that is distinct from jailbreaks or prompt injection, and harder to patch.

  Jul 3, 2026 rlhf reward-hacking alignment sycophancy ai-safety goodharts-law

• Jailbreak Robustness After Fine-Tuning: How Safety Alignment Degrades

  Safety fine-tuning instills refusal behaviors in LLMs — but those behaviors are surprisingly brittle under subsequent fine-tuning. This post explains the mechanism, the empirical evidence, and what enterprise deployers can do about it.

  Jun 30, 2026 fine-tuning safety-alignment jailbreaks rlhf threat-model defender-checklist

• Specification Gaming and Reward Hacking: When AI Optimizes for the Wrong Goal

  Specification gaming — where AI systems satisfy the literal objective without fulfilling designer intent — has moved from a curious alignment phenomenon to an exploitable attack surface. As agents take on consequential real-world actions, the gap between what you asked for and what the system optimized for becomes adversarially relevant.

  Jun 28, 2026 alignment reward-hacking agentic-ai threat-modeling specification-gaming rlhf

• Beyond Reward Hacking: RLHF's Spurious-Correlation Problem — and a Causal Fix

  Wang et al. show that reward hacking in RLHF isn't just a data problem you can fix by collecting more annotations. Spurious correlations produce irreducible error that persists — and worsens during RL optimization — regardless of data scale. Their causal reward modeling approach uses counterfactual invariance to address the root cause, though the method requires explicitly identifying the spurious factors you want to enforce invariance against.

  Jun 21, 2026 alignment rlhf reward-hacking threat-modeling training-security

• Alignment Tampering: How RLHF's Own Design Amplifies Bias

  A paper accepted at ICML 2026 demonstrates a structural vulnerability in on-policy RLHF pipelines where the model being trained influences its own preference dataset — causing alignment to amplify rather than suppress certain misaligned behaviors.

  Jun 20, 2026 alignment rlhf threat-modeling training-security bias