Self-hosted alternatives to eval-harness-first

E

eval-harness-first

Build the evaluation harness that gates every fine-tuning run

eval-harness-first is an agent skill from the wshobson/agents collection for building the evaluation harness that gates a fine-tuning run. It is the first phase of a fine-tuning workflow: the other skills in that sequence assume the harness exists before any training configuration is written. The harness is treated as more than a report at the end of a run. It is the engine for data curation, because the same labeled traces that build the golden test sets can feed the training data, with an explicit holdout kept back for testing. The skill covers golden sets of expected behavior, graders that check each failure mode separately, calibration of an LLM judge against human labels, and baselines measured on the base model before any fine-tuning, so a later result can be compared with where the model started. Its inputs are production or agent traces when they exist, or a task specification when they do not, plus labelers willing to grade at least 100 examples. Its output is an eval directory with goldens, graders, a drift suite and the base-model baseline that later phases check against. It is installed with the skills CLI from the wshobson/agents repository. Its skills.sh page showed about 2.4K installs and passing results from three security audits at the time of the scan.