Description
vision-sft is an agent skill from the wshobson/agents collection for supervised fine-tuning of vision-language models (VLMs), which take images and text together. It is used when a model needs adapting to a visual domain or task, and the training data is image and text demonstrations. The skill is part of the wshobson fine-tuning sequence. The routing step is assumed to have chosen supervised fine-tuning on image and text data and a vision-language base model, rather than preference pairs, a verifiable reward, or a text-only model. It specializes the text-only LoRA and QLoRA recipe for the parts of a vision-language model that process images, the vision tower and the projector that connects it to the language model. The lora-qlora-recipes skill covers the LoRA fundamentals such as rank, alpha and target modules, and should be read first if they are not familiar. The skill covers configuring the training run for these parts of the model. It applies when adapting a VLM to a visual domain, configuring a supervised run on image and text data, and deciding which parts of the model to train. As with other fine-tuning, the results should be measured against a baseline on images the model has not seen, to confirm the adaptation helped. The skill is installed with the skills CLI from the wshobson/agents repository. Its skills.sh page showed about 2.3K installs and passing results from three security audits at the time of the scan.…
Highlights
- Supervised fine-tuning of vision-language models
- Image and text demonstration data
- Vision tower and projector configuration
- Builds on the LoRA and QLoRA recipe
- Part of the fine-tuning sequence
- Installs with the skills CLI
Ratings & Reviews
No ratings yet
Be the first to rate vision-sft.
People also visit
finetuning-method-selection
Decide whether to fine-tune at all, and route to the right method
vision-sft FAQ
What kind of model does it target?
Vision-language models that take images and text together.
What data does it expect?
Image and text demonstrations for supervised fine-tuning.
Should I read another skill first?
Yes, if LoRA basics such as rank, alpha and target modules are unfamiliar. The lora-qlora-recipes skill covers them.
Which parts of the model does it cover?
The vision tower and projector, which the skill specializes from the text-only LoRA recipe.
How do I install it?
Run npx skills add https://github.com/wshobson/agents --skill vision-sft.
What security audits does it have?
skills.sh lists three audits: Gen Agent Trust Hub Pass, Socket Pass and Snyk Pass.
What are the best vision-sft alternatives?
Popular alternatives to vision-sft include lora-qlora-recipes, fine-tuning-expert and dataset-curation. See all vision-sft alternatives