preference-optimization
Align a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO
by wshobsonFree
Description
preference-optimization is an agent skill from the wshobson/agents collection for aligning a fine-tuned language model with preference data, using methods from the DPO family: DPO, ORPO, KTO and SimPO. Preference data comes either as pairs, where a reviewer picks which of two responses is better, or as unpaired thumbs-up and thumbs-down feedback. The skill applies when that kind of feedback exists and the model should learn from it rather than from demonstrations or a verifiable reward. Within the wshobson fine-tuning sequence, the routing step is assumed to have already sent the work here, and the other skills cover demonstration-based training and reward-based reinforcement learning. The skill covers the choice of method within the DPO family, the evidence on how much that choice actually changes results, the production training pattern, and how to build the preference pairs in the first place. Preference data is only as good as the judgments behind it, so the skill's guidance on building pairs matters as much as the training configuration. The skill is installed with the skills CLI from the wshobson/agents repository. Its skills.sh page showed about 2.4K installs and passing results from three security audits at the time of the scan.…
Highlights
- DPO, ORPO, KTO and SimPO methods
- Guidance on choosing within the DPO family
- Production training pattern
- Building preference pairs from feedback
- Part of the fine-tuning sequence
- Installs with the skills CLI
Ratings & Reviews
No ratings yet
Be the first to rate preference-optimization.
People also visit
finetuning-method-selection
Decide whether to fine-tune at all, and route to the right method
preference-optimization FAQ
What data does it need?
Preference pairs, where one response is judged better than another, or unpaired thumbs-up and thumbs-down feedback.
Which methods are covered?
DPO, ORPO, KTO and SimPO.
How are the pairs built?
The skill covers building the preference pairs, which matters as much as the training configuration.
What if I only have demonstrations?
That case is handled by the lora-qlora-recipes skill, and reward-based training by grpo-rlvr-training.
How do I install it?
Run npx skills add https://github.com/wshobson/agents --skill preference-optimization.
What security audits does it have?
skills.sh lists three audits: Gen Agent Trust Hub Pass, Socket Pass and Snyk Pass.
What are the best preference-optimization alternatives?
Popular alternatives to preference-optimization include grpo-rlvr-training, lora-qlora-recipes and dataset-curation. See all preference-optimization alternatives