OpenAI Details ChatGPT’s 3-Stage Post-Training Pipeline With 13,000 Prompts and RLHF
Updated
Updated · O'Reilly Media · Sep 21
OpenAI Details ChatGPT’s 3-Stage Post-Training Pipeline With 13,000 Prompts and RLHF
3 articles · Updated · O'Reilly Media · Sep 21
Summary
OpenAI said ChatGPT’s original post-training stack had three layers: supervised fine-tuning, a reward model, and RLHF—turning base GPT-3 from erratic text generation into a conversational assistant.
About 40 labelers wrote ideal answers for roughly 13,000 prompts in the SFT stage, while OpenAI mixed in 10% pretraining data and later used 31,000 user prompts to drive RL rollouts.
A 6B-parameter reward model learned from rankings of 4–9 responses per prompt, yielding about 200,000 to 1.2 million pairwise comparisons to score outputs humans preferred.
PPO-based RLHF then optimized the policy against that reward model while adding a KL-divergence penalty to keep it close to the SFT model and reduce reward hacking.
OpenAI said the approach improved helpfulness, safety, tool use and reasoning; in cited tests, even a 1.3B RLHF model often beat a 175B SFT model, though gains were smaller with held-out evaluators.