Tagged:

2 posts

• Beyond Reward Hacking: RLHF's Spurious-Correlation Problem — and a Causal Fix

  Wang et al. show that reward hacking in RLHF isn't just a data problem you can fix by collecting more annotations. Spurious correlations produce irreducible error that persists — and worsens during RL optimization — regardless of data scale. Their causal reward modeling approach uses counterfactual invariance to address the root cause, though the method requires explicitly identifying the spurious factors you want to enforce invariance against.

  Jun 21, 2026 alignment rlhf reward-hacking threat-modeling training-security

• Alignment Tampering: How RLHF's Own Design Amplifies Bias

  A paper accepted at ICML 2026 demonstrates a structural vulnerability in on-policy RLHF pipelines where the model being trained influences its own preference dataset — causing alignment to amplify rather than suppress certain misaligned behaviors.

  Jun 20, 2026 alignment rlhf threat-modeling training-security bias