[D] Starting the Training Journey: An Open-Source RLHF Full-Scale Training Framework for Building 70B+ Models Based on Ray and vLLM

Follow the full discussion on Reddit.
ChatGPT has been around for over a year now, and RLHF training is an indispensable part of training ChatGPT. Currently, there are already quite a few open-source RLHF training frameworks such as TRL, DeepSpeedChat or the recently popular LLaMA Factory. These frameworks are often based on parallelization methods like ZeRO, slicing the four models in the RLHF algorithm and placing them on the same GPU. In today's era of ever-larger model sizes, such scheduling cannot meet the needs of full-scale RLHF training for 70B+ or even just 13B+ models. It requires compromising on memory usage through merging the Actor Critic models or using methods like LoRA. However, these PEFT methods often mean compromising model performance.