Reward Over-Optimization

Reinforcement learning from human feedback (RLHF) is an effective method for aligning large language models (LLMs) with human values. …