evaluation-methodology
PluginEval quality methodology, covering dimensions, rubrics
by wshobsonFree
Description
evaluation-methodology is an agent skill from the wshobson/agents collection that explains how PluginEval measures the quality of a skill or plugin. PluginEval scores a skill or plugin from 0 to 100 by combining up to three layers of evaluation. The static layer is a lint: it is fast and deterministic and is useful for checking the structure of a skill. The LLM judge and Monte Carlo layers are more involved, and the skill is clear that both are experimental and not validated against human labels, so their numbers should be treated as rough signals rather than measurements. The skill is for understanding how plugin quality is measured, reading a score and its dimensions, and deciding how to act on a result. It points to the rubric anchors used by the judge, and to a reference on fixing each anti-pattern and improving each dimension of the score. For the trace-based evaluation program, it refers to the evals documentation at the root of the repository. It is useful for skill authors who want to know what a score is made of before they change their skill to raise it. It is installed with the skills CLI from the wshobson/agents repository. Its skills.sh page showed about 7.0K installs and passing results from three security audits at the time of the scan.…
Highlights
- PluginEval scoring from 0 to 100
- Static lint layer for structure
- LLM judge and Monte Carlo layers, marked experimental
- Rubric anchors for each dimension
- Guidance for improving scores
- Installs with the skills CLI
Ratings & Reviews
No ratings yet
Be the first to rate evaluation-methodology.
People also visit
evaluation-methodology FAQ
What is PluginEval?
A method for scoring the quality of a skill or plugin from 0 to 100.
How is the score built?
From up to three layers: a static lint, an LLM judge, and a Monte Carlo layer. The skill notes that the last two are experimental.
Can I trust the LLM judge numbers?
The skill says they are not validated against human labels, so treat them as rough signals.
Where are the rubrics?
The judge rubric anchors are in references/rubrics.md, and tips for each dimension are in references/improving-scores.md.
Who is it for?
Authors of skills and plugins who want to understand how quality is measured before they change their work.
How do I install it?
Run npx skills add https://github.com/wshobson/agents --skill evaluation-methodology.
What security audits does it have?
skills.sh lists three audits: Gen Agent Trust Hub Pass, Socket Pass and Snyk Pass.
What are the best evaluation-methodology alternatives?
Popular alternatives to evaluation-methodology include ai-debt-detector, code-review-excellence and eval-harness-first. See all evaluation-methodology alternatives