Self-hosted alternatives to trace-to-training-data

T

trace-to-training-data

Convert evaluation traces and production logs into SFT examples

trace-to-training-data is an agent skill from the wshobson/agents collection for converting graded evaluation traces and production logs into training data. Its job is the conversion step in a fine-tuning flywheel: the traces it works on have already been graded by an evaluation harness, so the labels exist, and this skill turns the good ones into supervised fine-tuning examples and the contrasting ones into preference pairs. The skill assumes the goldens, the graders and the run results from the eval-harness-first skill are in place before conversion starts, which keeps grading and training from being mixed up. The idea it carries is that the same labeled traces that test a model can become its training set, provided a holdout is kept back so the evaluation stays honest. The skill applies when graded traces or failure examples already exist and need to become training data. Conversion involves judgment calls about which traces to keep, how to pair good and bad responses, and what to hold back for testing, so the outputs should be reviewed before a run. Converted traces can carry personal or confidential content from production, so they should be filtered before they leave the systems where they were recorded. The skill is installed with the skills CLI from the wshobson/agents repository. Its skills.sh page showed about 2.4K installs, with Gen Agent Trust Hub and Socket passing and Snyk showing a warning at the time of the scan.