Developers Build 3-Stage Qwen2.5-1.5B Post-Training Pipeline With 4 GPUs
Updated
Updated · O'Reilly Media · Oct 7
Developers Build 3-Stage Qwen2.5-1.5B Post-Training Pipeline With 4 GPUs
1 articles · Updated · O'Reilly Media · Oct 7
Summary
Qwen2.5-1.5B is used in a hands-on walkthrough that implements the classic post-training stack end to end: supervised fine-tuning, reward-model training and PPO.
Stage 1 fine-tunes the base model with torchtune on conversational JSONL data, using LoRA, a 2e-5 learning rate and 3 epochs to teach instruction-following rather than prompt completion.
Stage 2 trains a reward model from the SFT checkpoint with TRL on chosen-versus-rejected response pairs, using 1 epoch and a 1e-5 learning rate because reward models overfit quickly.
Stage 3 runs PPO in verl with prompts in parquet format, a 1e-6 actor learning rate, 0.1 KL penalty and 500 training steps, while monitoring reward, KL drift and sampled outputs.
The guide says the full pipeline can run on a single node with at least 2 GPUs and about 80 GB of GPU memory, though 4 or 8 GPUs are preferred.