eval-harness-first
Build the evaluation harness that gates every fine-tuning run
by wshobsonFree
Description
eval-harness-first is an agent skill from the wshobson/agents collection for building the evaluation harness that gates a fine-tuning run. It is the first phase of a fine-tuning workflow: the other skills in that sequence assume the harness exists before any training configuration is written. The harness is treated as more than a report at the end of a run. It is the engine for data curation, because the same labeled traces that build the golden test sets can feed the training data, with an explicit holdout kept back for testing. The skill covers golden sets of expected behavior, graders that check each failure mode separately, calibration of an LLM judge against human labels, and baselines measured on the base model before any fine-tuning, so a later result can be compared with where the model started. Its inputs are production or agent traces when they exist, or a task specification when they do not, plus labelers willing to grade at least 100 examples. Its output is an eval directory with goldens, graders, a drift suite and the base-model baseline that later phases check against. It is installed with the skills CLI from the wshobson/agents repository. Its skills.sh page showed about 2.4K installs and passing results from three security audits at the time of the scan.…
Highlights
- Evaluation harness before any fine-tuning run
- Golden sets and per-failure-mode graders
- Judge calibration against human labels
- Base-model baselines
- Holdout split shared with training data
- Installs with the skills CLI
Ratings & Reviews
No ratings yet
Be the first to rate eval-harness-first.
People also visit
eval-harness-first FAQ
What does this skill produce?
An eval directory with golden sets, graders, a drift suite and the base-model baseline.
What do I need to start?
Production or agent traces if you have them, or a task specification if you do not, plus labelers who will grade at least 100 examples.
Why measure the base model?
So fine-tuning results can be compared with where the model started, and checked against the baseline in later phases.
What does judge calibration mean?
Checking an LLM judge against human labels before trusting its grades.
Is it part of a larger workflow?
Yes. It is the first phase of the wshobson fine-tuning sequence, which the other skills assume is in place.
How do I install it?
Run npx skills add https://github.com/wshobson/agents --skill eval-harness-first.
What security audits does it have?
skills.sh lists three audits: Gen Agent Trust Hub Pass, Socket Pass and Snyk Pass.
What are the best eval-harness-first alternatives?
Popular alternatives to eval-harness-first include eval-harness, skill-comply and google-agents-cli-eval. See all eval-harness-first alternatives