RL from Human Feedback

- Overview

Reinforcement Learning (RL) from Human Feedback (RLHF) is a machine learning (ML) technique that aligns AI models - especially large language models (LLMs) like ChatGPT or Claude - with human values, safety, and intentions. 

Instead of relying purely on raw internet text or rigid, hand-coded rules, RLHF uses human judgments to guide how a model learns and responds. 

1. How RLHF Works (Typical Steps):

• Pretraining & Supervised Fine-Tuning (SFT): A base model is first trained on a massive text corpus, then fine-tuned with human-labeled examples to learn the expected conversation format. 
• Reward Model Training: Human evaluators compare multiple model responses to a prompt and rank them. A separate "reward model" is trained on this ranking data to output a scalar reward score predicting what humans prefer. 
• Policy Optimization: An optimization algorithm—most commonly Proximal Policy Optimization (PPO)—uses the reward model's score to update the agent's policy while preventing it from drifting too far from its original behavior (often using a Kullback–Leibler divergence penalty). 

2. Common Applications:

• Natural Language Processing (NLP): Conversational assistants and text summarizers (e.g., ChatGPT, Claude).
• Computer Vision: Text-to-image generation models.
• Robotics & Gaming: Training video game bots and physical agents to perform complex, subjective tasks.

3. Key Limitations & Challenges:

• Cost & Labor: Sourcing high-quality human preference data is time-consuming and expensive.
• Human Biases & Inconsistency: Evaluators can bring subjective viewpoints, cultural biases, or errors into the ranking data.
• Reward Hacking: The agent may exploit flaws in the reward model to score high rewards without actually fulfilling the true human intent.
• Alternatives: Newer direct-alignment methods skip the separate reward model and RL loop entirely, such as Direct Preference Optimization (DPO), IPO, and KTO.

Please refer to the following for more information:

• Wikipedia: Reinforcement Learning (RL) from Human Feedback (RLHF)

- How RLHF Works: The 3-Stage Pipeline

The classic RLHF workflow converts human intuition into an automated training signal through three main steps:

1. Supervised Fine-Tuning (SFT):

• A pre-trained base model is trained on a curated dataset of high-quality prompts and human-written demonstration responses.

• This teaches the model the basic format and style of a helpful conversational assistant.

2. Reward Model Training:

• The SFT model generates multiple responses to the same prompt.

• Human evaluators review and rank these responses (e.g., choosing which answer is more helpful, honest, or safe).

• A separate reward model is trained on this ranking data to predict scalar score values that mimic human preference judgments. 

3. Policy Optimization (Reinforcement Learning):

• Using an algorithm like Proximal Policy Optimization (PPO), the language model is updated to maximize the reward score it gets from the reward model.
• A mathematical penalty (called KL divergence) is applied to keep the model from drifting too far from its original behavior or "hacking" the reward system with nonsense text.

- Why RLHF is Needed

Reinforcement Learning from Human Feedback (RLHF) is a machine learning technique used to train and align intelligent agents (such as large language models) with human values, intentions, and safety standards. 

• Fixing Raw Predictions: Base models are only trained to predict the "next word" from massive datasets, which can lead to hallucinations, bias, or ignored instructions.
• Subjective Quality: Many desirable traits—like tone, helpfulness, and safety—are difficult or impossible to write down as a simple math formula.
• User Alignment: RLHF bridges the gap between raw statistical text generation and what humans actually consider a good response.

- Limitations and Alternatives

(A) High Cost & Labor: 

• Collecting human rankings and writing responses is expensive, time-consuming, and prone to human annotator fatigue or bias.

(B) Reward Hacking: 

• The model may find exploits to game the reward model without actually improving quality. 

(C) Emerging Alternatives:

• RLAIF (Reinforcement Learning from AI Feedback), where a stronger AI judges responses instead of humans.
• Direct alignment methods like DPO (Direct Preference Optimization), IPO, and KTO that skip the explicit reinforcement learning loop.

[More to come ...]