Swap the evaluated agent in one line
Keep task logic unchanged: install a plugin, update one line in config, and run the same dataset across configurations.
Configurable and pluggableagent evaluation framework
Compose environments and evaluated agents with plugins. Equip coding agents with the CLI and skills to run evaluations end to end.
Automate evaluation with coding agents
Install the CLI and skills so a coding agent can design, convert, and run benchmarks end to end.
Share and reuse on Hub
Publish datasets, plugins, Agent packages, and traceable evaluation results on ageval Hub.
// 01The Problem
Evaluation cannot compare models alone.
What executes tasks is Model + Harness.
01 / MODEL
Swapping only the model ID
Prompts, tools, and interaction loops stay fixed while model B replaces model A to produce a single number.
02 / HARNESS
The full agent system does the work
Task outcomes depend on the model together with the harness managing the interaction loop, tool calls, and execution state.
03 / DRIFT
Changing harness or environment shifts scores
The same model paired with a different harness or execution environment yields different pass rates and resource costs.
04 / RECORD
Unlocked configurations cannot be compared
Reporting a bare model name leaves the harness, model parameters, and environment unspecified, making scores non-comparable.
SOLUTION → ageval composes environments and agent executors through plugins and locks the full configuration before execution, making scores reproducible and comparable
// 02Positioning
Keep Core fixed. Swap components with plugins.
ageval Core coordinates a fixed pipeline: lock → environment → run → evaluate → record. Execution environments and agent executors connect via declarative plugin slots without modifying Core or task scripts.
// INPUTS User commands, dataset definitions, and profiles enter from outside. The dataset defines task logic and grading criteria; environment and role bindings live externally in profiles.yaml.
// 03Environment
Swap environments. Keep the dataset.
Run the same task on your host machine, in a local container, in a cloud sandbox, or over SSH by changing one line in profiles.yaml.
ENV · LOCAL
Host directory
Real filesystem with subprocess execution.
RUNS TODAY
ENV · DOCKER
Local container
Container isolation; reference materials (gold) are uploaded only during evaluate.
RUNS TODAY
ENV · E2B / SSH / DAYTONA
Cloud sandboxes and remote hosts
Local orchestration with remote execution once the provider SDK and credentials are configured.
NEEDS CREDENTIALS
// 04Auto
Teach coding agents automated evaluation.
Install the CLI and skills so a coding agent can design, convert, and run benchmarks end to end. Load only the skills your task requires.
| Skill | What it does |
|---|---|
ageval-platform | Architecture overview: ownership boundaries, configuration locking, and evaluation rules. |
ageval-cli | Run lock / run / campaign; inspect, regrade, and upload results. |
ageval-config-package | Author or convert a dataset: ageval.yaml, task.yaml, and profiles.yaml. |
ageval-sdk-harness | Write task scripts: AgentSession, tools, and artifact publishing. |
ageval-plugin | Develop plugins: environment providers and agent executors. |
// 05Plugin
Compose environments and agents with plugins.
Swap the evaluated agent or environment by installing a plugin and updating profiles.yaml — with zero changes to ageval Core.
ENV
Environment plugins
Host / Docker / cloud sandbox / remote SSH. Implement the unified EnvironmentProvider protocol to add a backend.
AGENT
Executor plugins
ACP by default (pi / Codex / Claude Code / OpenCode); nooa / dsh / miniswe connect through the same executor slot.
LOCK
Static verification at lock
Unmatched capability requirements, model parameters, or credential locators fail fast during lock before execution begins.
EXAMPLE · DSH
DeepSeek Harness connects as an executor plugin — zero lines changed in ageval Core.
Although dsh uses its own JSON-RPC runtime rather than ACP, it plugs into the same executor slot: install the extra, set executor to dsh in profiles, and run the same task. Missing credentials fail fast at lock/preflight.
# Install the dsh plugin extra uv tool install 'ageval-cli[dsh]' # Switch executor to dsh; task and dataset remain untouched ageval run <dataset> --task <task-id> \ --profiles profiles.dsh.yaml # executor: dsh
// 06FAQ
FAQ
How do I switch the evaluated agent?
ageval evaluates the combined agent system (Model + Harness, such as pi, Codex, Claude Code, or dsh) rather than a bare model ID. Install the corresponding plugin and update profiles.yaml to run the same dataset without modifying task code.
Built-in Agent packages bind directly with --agent pi (and optional --model). Custom Agent packages install via ageval agent install and bind with --agent org/name@version.
After installing the CLI, how does my coding agent run evaluations?
Run uv tool install ageval-cli and npx skills add ZJU-REAL/ageval. The skills teach your coding agent how to invoke lock / run and author datasets, allowing it to convert benchmarks and execute evaluations end to end.
Live coding-agent runs require the corresponding ACP entry and credentials on the host, whereas ageval lock validates configuration without starting an agent. Missing extras or credentials fail preflight with an explicit install hint.
How do I run a dataset from Hub and share results?
Use ageval registry list to inspect available datasets, then run ageval run <org>/<name>@<version> (or pass a local dataset root). Default profiles use environment: docker, which requires a running Docker engine.
Publish datasets, plugins, Agent packages, and suite results to ageval Hub for reproducible comparison, inspect runs locally with ageval view, or generate a read-only /s/{token} snapshot link.
What files belong in a dataset?
A dataset is the standard delivery unit: ageval.yaml sits at the root, and each task lives under tasks/<id>/ with task.yaml, interaction logic in run.py, grading logic in evaluator.py, input files in data/, and reference materials in evaluation/. Environment and role bindings stay external in profiles.yaml.
Task scripts own the interaction loop and grading criteria, while ageval Core manages configuration locking, environment provisioning, verdict binding, and finally cleanup. The Python SDK never decides PASS or holds plaintext host secrets.
How do I author a custom environment or agent plugin?
Create an ageval.plugin/1 package that implements the environment or executor exclusive slot and declares its required service capabilities via inject. Once installed, select it in profiles.yaml without forking ageval.
If declared capability dependencies or model parameters are unsatisfied, ageval lock fails fast before execution starts.
Do I need to rewrite the dataset to switch among host, Docker, and cloud sandboxes? Can the solver see reference materials?
No rewrite is needed. Switch environment in profiles.yaml among local, docker, e2b, ssh, or daytona to run the same task across backends.
Reference materials (gold) live under tasks/<id>/evaluation/, remain unmounted during the run phase, and are uploaded to the scoring environment only when evaluate begins.