Self-hosted alternatives to autoresearch

A

autoresearch

Autonomous iterative experimentation loop for any programming task

autoresearch is an agent skill from the github/awesome-copilot collection for running an autonomous experimentation loop on any programming task that has a measurable outcome. The skill is inspired by Karpathy's autoresearch and generalizes it from machine learning training to any task. The user first defines the goal, the metric that measures it, and the scope constraints that limit what may change. The agent then iterates on its own: it modifies code, runs an experiment, measures the result, and keeps the change if it improved the metric or discards it if it did not, repeating until the user interrupts. The skill sets out agent behavior rules that keep the loop disciplined, such as changing only what the scope allows and recording each experiment so the history of what was tried is clear. It is useful for tuning performance, improving a model's score on a test, or reducing the size of a build, wherever the result can be measured automatically. The loop can run for a long time, so the metric must be trustworthy, and the scope should keep the agent away from code that must not change. Because the agent is changing code without a person reviewing each step, the kept changes should be reviewed before they are merged. The skill is installed with the skills CLI from the github/awesome-copilot repository. Its skills.sh page showed about 3.4K installs and passing results from three security audits at the time of the scan.