Reinforcement learning from human feedback

Reinforcement learning from human feedback, or RLHF, uses human judgments of model outputs as a learning signal. One well-known pipeline trains a reward model from ranked responses and then optimizes the response model against that reward.¹

Tiny example

Reviewers prefer answer A over answer B for many prompts. A model learns to predict those preferences, and training shifts the generator toward outputs that score better.

FDE note

RLHF is a model-development technique, not an application-time feedback button. It does not replace retrieval, authorization, or evaluation, and “human preference” depends on the people, rubric, and data collected.

Footnotes

1. Ouyang et al., “Training language models to follow instructions with human feedback”, 2022. ↩