Self-hosted alternatives to evaluation-methodology

E

evaluation-methodology

PluginEval quality methodology, covering dimensions, rubrics

evaluation-methodology is an agent skill from the wshobson/agents collection that explains how PluginEval measures the quality of a skill or plugin. PluginEval scores a skill or plugin from 0 to 100 by combining up to three layers of evaluation. The static layer is a lint: it is fast and deterministic and is useful for checking the structure of a skill. The LLM judge and Monte Carlo layers are more involved, and the skill is clear that both are experimental and not validated against human labels, so their numbers should be treated as rough signals rather than measurements. The skill is for understanding how plugin quality is measured, reading a score and its dimensions, and deciding how to act on a result. It points to the rubric anchors used by the judge, and to a reference on fixing each anti-pattern and improving each dimension of the score. For the trace-based evaluation program, it refers to the evals documentation at the root of the repository. It is useful for skill authors who want to know what a score is made of before they change their skill to raise it. It is installed with the skills CLI from the wshobson/agents repository. Its skills.sh page showed about 7.0K installs and passing results from three security audits at the time of the scan.