RLHF
RLHF (Reinforcement Learning from Human Feedback) is a method for training AI models that collects human preference feedback on model outputs to guide the model toward generating responses that better align with human values and expectations.
How it works
It works by having the AI model generate multiple answers, which human annotators rank; the preference data are then used to train a "reward model", and a reinforcement learning algorithm finally optimizes the original model so its outputs earn higher rewards (i.e., are more favored by humans). This approach is essential for aligning powerful foundation models (such as large language models) with complex, ambiguous human preferences.
Examples
- 1.Training chatbots like ChatGPT so their answers are more helpful, harmless, and honest.
- 2.Improving AI writing assistants so the text they generate better matches users' specific requirements in style and content.
- 3.Tuning code generation models so the code they produce is safer, more efficient, and better commented.