Description
eval-driven-dev is an agent skill from the github/awesome-copilot collection for building an evaluation pipeline that tests a Python LLM application end to end. The pipeline runs the application the way a real user would, with real inputs, and scores the outputs with evaluators such as LLM-as-judge and similarity scores, producing pass or fail results through the pixie test runner. Because LLM outputs vary, the tests use evaluators rather than exact comparisons. The skill is precise about what is under test: the application's own code, including request handling, context assembly, routing and response formatting. The language model must be real and must not be mocked or replaced, while the data the application reads from databases, caches or third-party APIs is supplied as test values through instrumentations, so each test controls exactly what the application sees. The skill also covers defining evaluation criteria, building golden datasets of expected cases, and observing and scoring the results over time. It is useful for teams building AI features who want regression tests that catch when a change makes the answers worse. Evaluation with an LLM judge costs model calls and its scores are approximate, so the criteria and a sample of human-checked results should be used to calibrate trust in the judge. The skill is installed with the skills CLI from the github/awesome-copilot repository. Its skills.sh page showed about 3.9K installs, with Socket and Snyk showing warnings and Gen Agent Trust Hub passing at the time of the scan.…
Highlights
- End-to-end evaluation of Python LLM applications
- LLM-as-judge and similarity evaluators
- Pass and fail results through pixie test
- Real language model calls, nothing mocked
- Instrumented external data for controlled tests
- Installs with the skills CLI
Ratings & Reviews
No ratings yet
Be the first to rate eval-driven-dev.
People also visit
canvas-design
Create beautiful visual art in .png and .pdf documents using design philosophy
eval-driven-dev FAQ
How are outputs scored?
With evaluators such as LLM-as-judge and similarity scores, since exact matching does not suit varied LLM output.
Is the model mocked?
No. The skill requires real LLM calls, and mocks only the external data sources the application reads.
What is an instrumentation here?
A way to supply test values for the data the application reads from databases, caches or third-party APIs.
Does LLM judging cost money?
It makes model calls, and its scores are approximate, so calibrate trust in the judge with human-checked results.
How do I install it?
Run npx skills add https://github.com/github/awesome-copilot --skill eval-driven-dev.
What security audits does it have?
skills.sh lists three audits: Gen Agent Trust Hub Pass, Socket Warn and Snyk Warn. A Warn or Fail is worth reading before you install.
What are the best eval-driven-dev alternatives?
Popular alternatives to eval-driven-dev include implement, brand-guidelines and canvas-design. See all eval-driven-dev alternatives