trace-to-training-data
Convert evaluation traces and production logs into SFT examples
by wshobsonFree
Description
trace-to-training-data is an agent skill from the wshobson/agents collection for converting graded evaluation traces and production logs into training data. Its job is the conversion step in a fine-tuning flywheel: the traces it works on have already been graded by an evaluation harness, so the labels exist, and this skill turns the good ones into supervised fine-tuning examples and the contrasting ones into preference pairs. The skill assumes the goldens, the graders and the run results from the eval-harness-first skill are in place before conversion starts, which keeps grading and training from being mixed up. The idea it carries is that the same labeled traces that test a model can become its training set, provided a holdout is kept back so the evaluation stays honest. The skill applies when graded traces or failure examples already exist and need to become training data. Conversion involves judgment calls about which traces to keep, how to pair good and bad responses, and what to hold back for testing, so the outputs should be reviewed before a run. Converted traces can carry personal or confidential content from production, so they should be filtered before they leave the systems where they were recorded. The skill is installed with the skills CLI from the wshobson/agents repository. Its skills.sh page showed about 2.4K installs, with Gen Agent Trust Hub and Socket passing and Snyk showing a warning at the time of the scan.…
Highlights
- Converts graded traces into SFT examples
- Builds preference pairs from contrasting traces
- Keeps a holdout for honest evaluation
- Works from the eval-harness-first outputs
- Covers production logs and failure examples
- Installs with the skills CLI
Ratings & Reviews
No ratings yet
Be the first to rate trace-to-training-data.
People also visit
finetuning-method-selection
Decide whether to fine-tune at all, and route to the right method
trace-to-training-data FAQ
What does this skill turn into what?
Graded evaluation traces and production logs become supervised fine-tuning examples and preference pairs.
What must exist before it runs?
The goldens, graders and run results from the eval-harness-first skill, so the traces are already graded.
Why keep a holdout?
So the evaluation stays honest, since traces used for training should not also be used to test the model.
Should I check the output?
Yes. Conversion involves judgment calls that should be reviewed before a training run.
How do I install it?
Run npx skills add https://github.com/wshobson/agents --skill trace-to-training-data.
What security audits does it have?
skills.sh lists three audits: Gen Agent Trust Hub Pass, Socket Pass and Snyk Warn. A Warn or Fail is worth reading before you install.
What are the best trace-to-training-data alternatives?
Popular alternatives to trace-to-training-data include dataset-curation, finetuning-method-selection and fine-tuning-expert. See all trace-to-training-data alternatives