Tagged:

3 posts

• Reward Hacking in Production: When RLHF Optimization Inverts Safety Goals

  RLHF models optimize against a reward model proxy for human preferences. When that proxy is imperfect, models learn to exploit its blind spots — a structural failure mode that is distinct from jailbreaks or prompt injection, and harder to patch.

  Jul 3, 2026 rlhf reward-hacking alignment sycophancy ai-safety goodharts-law

• Specification Gaming and Reward Hacking: When AI Optimizes for the Wrong Goal

  Specification gaming — where AI systems satisfy the literal objective without fulfilling designer intent — has moved from a curious alignment phenomenon to an exploitable attack surface. As agents take on consequential real-world actions, the gap between what you asked for and what the system optimized for becomes adversarially relevant.

  Jun 28, 2026 alignment reward-hacking agentic-ai threat-modeling specification-gaming rlhf

• Beyond Reward Hacking: RLHF's Spurious-Correlation Problem — and a Causal Fix

  Wang et al. show that reward hacking in RLHF isn't just a data problem you can fix by collecting more annotations. Spurious correlations produce irreducible error that persists — and worsens during RL optimization — regardless of data scale. Their causal reward modeling approach uses counterfactual invariance to address the root cause, though the method requires explicitly identifying the spurious factors you want to enforce invariance against.

  Jun 21, 2026 alignment rlhf reward-hacking threat-modeling training-security