aiaimodel.cards
Weights & training53

RLHF

Training on human preferences between responses

RLHF

Reinforcement learning from human feedback: people compare model responses, a reward model learns their preferences, and the model is trained to produce responses that score higher.

Why it matters. It is a main way assistants are made helpful and harmless, and it explains traits such as verbosity, over-refusal and sycophancy.

How it works

Classic RLHF has three steps: collect human judgments of which of two responses is better, train a reward model to predict those judgments, then optimize the language model against the reward model with reinforcement learning, with a penalty for drifting too far from the starting model.

Variants include AI feedback guided by written principles instead of human ratings, direct preference methods such as DPO that skip the separate reward model and RL loop, and reinforcement learning with automatically checkable rewards for math and code. Makers combine these and describe them in their cards at different levels of detail.

Preference training can overshoot: models learn what raters reward, such as long, confident or agreeable answers. When a model is too verbose, too cautious or too agreeable, the cause often lies here, and prompting or provider settings are the practical fix.

Worked example

Illustrative: 100,000 comparisons at 30 seconds each take 100,000 x 30 / 3,600 = about 833 hours of rating. Having three people rate each comparison to check agreement raises that to about 2,500 hours.

Common mistakes

  • Reading "trained with RLHF" as a guarantee of safety or accuracy; it shapes preferences and style, not truth.
  • Mistaking confident, well-formatted answers for correct ones, a style preference training tends to reward.
  • Assuming every maker means the same process by RLHF; methods and feedback sources differ.

Related terms