RLHF: How AI Learns from Human Feedback

When OpenAI finished training GPT-3, it wasn’t a chatbot. It was a completion engine. If you typed: “The capital of France is” It might reply: “usually sunny in the spring.” (Because it’s just predicting the next likely words).

To turn a raw text predictor into a helpful assistant (ChatGPT), they used a technique called RLHF (Reinforcement Learning from Human Feedback).

Step 1: Supervised Fine-Tuning (SFT)

First, humans write thousands of ideal conversations.

• Human: “How do I make a cake?”
• Human Writer: “Here is a recipe: 1. Flour, 2. Sugar…”

The model is trained on these to learn the format of a dialogue. But this is expensive (humans can’t write millions of answers).

Step 2: Reward Modeling (The Judge)

The model generates multiple answers to a prompt. Humans rank them.

• Prompt: “Explain the moon landing to a 6-year-old.”
• Response A: “The Apollo 11 mission utilized a Saturn V rocket…” (Too complex)
• Response B: “Neil Armstrong flew a big spaceship to the moon!” (Good)

The human prefers B > A. This data trains a second AI model called the Reward Model. Its only job is to look at text and predict: “How much would a human like this?”

Step 3: PPO (Proximal Policy Optimization)

Now the magic happens. We let the AI run wild, generating millions of answers. Instead of a human grading them, the Reward Model grades them.

• If the Reward Model gives a high score, the main AI is updated to do more of that.
• If the score is low, it’s updated to do less of that.

This is Reinforcement Learning. The AI is “playing the game” of conversation, trying to maximize its high score from the Reward Model.

Why RLHF matters

RLHF is what creates the “personality” of the AI.

• It teaches it to be Helpful (answer the question).
• It teaches it to be Harmless (refuse to build bombs).
• It teaches it to be Honest (try not to hallucinate, though this is hard).

Without RLHF, LLMs would just be uncensored, chaotic autocomplete engines. With RLHF, they become products.