agent-platform-eval-flywheel
Measures and improves the quality of AI models and agents on Google Cloud
by googleFree
Description
Agent Platform eval flywheel is an agent skill for measuring and improving the quality of generative AI models and agents on Google Cloud. It uses the Agent Platform GenAI Evaluation SDK, through client.evals.evaluate(), and follows what the skill calls the Eval Quality Flywheel, a loop in which you evaluate, read the results, change the system and evaluate again. The skill covers creating evaluation datasets from session traces, from pandas DataFrames or by synthetic generation, selecting and configuring evaluation metrics and writing custom ones, and analyzing rubric verdicts, loss patterns and clusters of failures. Based on the results, it suggests concrete code or prompt changes. It can also evaluate a model served on an Agent Platform endpoint, a bring-your-own model, or a Model-as-a-Service model by ID, and it will deploy the model first when needed, using the deployment reference and the endpoint and MaaS evaluation scripts it includes. The skill is useful for teams that want an evidence-based way to tell whether a model or agent change made things better or worse. It is published by Google on skills.sh under the google skills repository, and the listing shows Gen Agent Trust Hub, Socket and Snyk all passing.…
Highlights
- Evaluates models and agents with the GenAI Evaluation SDK
- Datasets from traces, DataFrames or synthetic data
- Custom and selected metrics
- Failure clustering and rubric analysis
- Evaluation for endpoint and MaaS models
Ratings & Reviews
No ratings yet
Be the first to rate agent-platform-eval-flywheel.
People also visit
google-agents-cli-eval
Evaluate ADK agents with metrics, evalsets, and the iterative eval-fix loop
agent-platform-eval-flywheel FAQ
What does the skill do?
It helps you measure and improve the quality of GenAI models and agents on Google Cloud, using the Agent Platform GenAI Evaluation SDK.
What data can it evaluate on?
Datasets built from session traces, pandas DataFrames, or synthetic user generation.
Can it evaluate a deployed model?
Yes. It covers models on an Agent Platform endpoint and Model-as-a-Service models, deploying the model first if needed.
Does it change my code?
It suggests concrete code or prompt improvements based on the eval results.
How do I install it?
Run npx skills add https://github.com/google/skills --skill agent-platform-eval-flywheel.
What are the best agent-platform-eval-flywheel alternatives?
Popular alternatives to agent-platform-eval-flywheel include agent-eval, google-agents-cli-eval and google-agents-cli-workflow. See all agent-platform-eval-flywheel alternatives