ORPO: Preference Optimization without the Supervised Fine-tuning (SFT) Step
A much cheaper alignment method but performing as well as DPO
There are now many methods to align large language models (LLMs) with human preferences. Reinforcement learning with human feedback (RLHF) was one of the first and brought us ChatGPT, but RLHF is very costly. DPO, IPO, and KTO are notably cheaper than RLHF as they don’t need a reward model.
While DPO and IPO are cheaper, they still require to train two different models. One model for the supervised fine-tuning (SFT) step, i.e., training the model to answer instructions, and then the model to align with human preferences using the SFT model for initialization and as a reference.
ORPO is yet another new method for LLM alignment but this one doesn’t even need the SFT model. With ORPO, the LLM jointly learns to answer instructions and human preferences.
In this article, I explain ORPO and review its performance. I show how to use it to turn Mistral 7B into a chat model using consumer hardware. ORPO is indeed cheaper than DPO and IPO as it requires less GPU memory.
The notebook showing how to align Mistral 7B (or any other LLMs) with ORPO is available here:
The notebook also includes an example of ORPO training with GaLore.
