Self-hosted alternatives to vision-sft

V

vision-sft

Fine-tune vision-language models

vision-sft is an agent skill from the wshobson/agents collection for supervised fine-tuning of vision-language models (VLMs), which take images and text together. It is used when a model needs adapting to a visual domain or task, and the training data is image and text demonstrations. The skill is part of the wshobson fine-tuning sequence. The routing step is assumed to have chosen supervised fine-tuning on image and text data and a vision-language base model, rather than preference pairs, a verifiable reward, or a text-only model. It specializes the text-only LoRA and QLoRA recipe for the parts of a vision-language model that process images, the vision tower and the projector that connects it to the language model. The lora-qlora-recipes skill covers the LoRA fundamentals such as rank, alpha and target modules, and should be read first if they are not familiar. The skill covers configuring the training run for these parts of the model. It applies when adapting a VLM to a visual domain, configuring a supervised run on image and text data, and deciding which parts of the model to train. As with other fine-tuning, the results should be measured against a baseline on images the model has not seen, to confirm the adaptation helped. The skill is installed with the skills CLI from the wshobson/agents repository. Its skills.sh page showed about 2.3K installs and passing results from three security audits at the time of the scan.