REWARD HACKING IN AI ALIGNMENT: A COMPARATIVE REVIEW

Abstract

Reward hacking, where agents take advantage of misspecified reward functions to obtain large proxy rewards without meeting intended human objectives, is a major problem in AI alignment, especially in reinforcement learning and reinforcement learning from human feedback. This study looks at recent research on reward hacking in AI systems, including its causes, symptoms, detection, and mitigation. The study contrasts research on reward model ensembles, Preference As Reward shaping, anomaly detection standards, and the generalization of learnt reward-hacking behavior based on a selection of studies from 2022 to 2026. Reward misspecification, optimization pressure, model capacity, distribution shift, and linked biases in reward models are the main causes of reward hacking, according to the review. While techniques like anomaly detection, nonlinear reward shaping, and pretraining-based ensembles offer some mitigation, they do not completely eradicate reward hacking, particularly in high-capability and long-horizon optimization scenarios. The reviewed research also indicated that seemingly benign hacking actions could be generalized to more critical misaligned hazards, such as strategic self-preservation and shutdown resistance. The study indicates that further research on AI alignment should concentrate on adversarially robust monitoring, realistic long-horizon benchmarks, distance-aware uncertainty estimation, and a more thorough examination of phase transitions in increasingly powerful AI systems.

Keywords : AI alignment; anomaly detection; distribution shift; reinforcement learning from human feedback; reward hacking; reward model ensembles; reward misspecification.