CLI reference
Commands, --task, Always-k, --set allowlist.
Trust ageval --help. On a TTY, commands print a short recap (no digest / sha256); a pipe still writes JSON. ageval run --json forces the full document.
ageval tasks <dataset>
ageval prepare <dataset> [--allow-prepare]
ageval lock <dataset> --task <id>
ageval lock <dataset> --task <id> --probe
ageval run <dataset> --task <id>
ageval run <dataset> --task <id> --probe
ageval run <dataset>
ageval run official/demo@0.1.0 --dir tmp --task <id>
ageval run <dataset> --max-concurrent-tasks 2
ageval run <dataset> -k 5
ageval run <dataset> --task <id> --resume <attempt_id>
ageval checkpoints <dataset> --run <attempt_id>
ageval replay <dataset> --run <attempt_id> [--evaluate]
ageval run <dataset> --task <id> --fork <attempt_id> --checkpoint <seq> --model <model_id>
ageval run <dataset> --resume-suite <suite_run_id> --task <id> -k 5
ageval run <dataset> --resume-suite <suite_run_id> --retry-errors
ageval results regrade <dataset> --run <attempt_id>
ageval campaign <dataset> --task <id> ...
ageval status <suite_run_id> --dataset <dataset>
ageval cancel <suite_run_id> --dataset <dataset>
ageval executors -v
ageval plugin install <path|ref>
ageval plugin list
ageval plugin uninstall <plugin_id>
ageval agent install <path|ref>
ageval agent list
ageval agent uninstall <agent_id>
ageval evidence --help
ageval view <dataset|directory|id@version> [--dev] [--open /jobs/<id>]
# Harbor task directory or task-set: view opens it and its .ageval/suite-runs.
# run and lock read that tree only with --layout harbor.
ageval jobs delete --local <dataset> --job <id> --yes
# Registry / results (login required)
ageval publish <dataset> --org <org> [--draft] [--public] [--replace]
ageval publish --repo github:org/repo@v1.0.0 [--public]
ageval release <dataset_id> [--public] [--replace]
ageval registry list|show|set-visibility|delete|org-create|org-list|org-add-member|org-set-role|org-transfer|org-remove-member …
ageval results upload-suite <dataset|id@version> --suite-run <8-hex>
ageval results analyze <dataset> (--run <id> | --suite-run <id>) [--analyzer stats|hack|review]
ageval results upload|attach-agent|request|inbox|decide|share|unshare|set-visibility|delete …
ageval results share-snapshot <dataset> --suite-run <id> [--run <run_id>] [--live]
ageval results sync-snapshot <dataset> --suite-run <id> [--heartbeat]
ageval results revoke-snapshot <token-or-url>Path = dataset root. run / lock / view / upload-suite also accept a Hub ref (id@version) or a pinned git source (github:org/repo@<tag|sha>, https://…@<tag|sha>, optional #<subdir> for one Dataset inside a repo); branch refs are rejected, and an https URL must not embed userinfo. --dir only with a Hub ref. tasks / campaign stay local directories. publish --repo <git ref> registers the source on the Hub without uploading files. run / lock first run dataset preparation (declared assets downloads, verified by sha256); a declared prepare script needs --allow-prepare. --agent and --profiles are mutually exclusive. --model requires --agent. ageval agent install also installs declared non-contrib plugins. Destructive commands require --yes; overwrite same identity requires --replace (default 409). AGEVAL_REGISTRY_URL / --registry-url override the Registry origin; omit them to use the public Hub (https://ageval.zjureal.com).
results analyze reads one sealed run (--run) or one suite (--suite-run) and writes analysis/<analyzer>/report.json. A suite also writes analysis/<analyzer>/summary.json. The default analyzer is stats. hack and review can call an existing openai-http or anthropic-http agent: pass --agent and --model (required when that package has no model). --base-url overrides that binding and must be an http(s) URL. This --model is not ageval run --model and not results upload-suite --model. Analysis does not change PASS or evaluation/checks.json, and Hub does not upload it.
share-snapshot prints {hub}/s/{token}. It does not change visibility or leaderboard listing. sync-snapshot updates a live link. See Hub.
With --layout harbor, the built-in run sends instruction.md once to role solver. That role id is fixed. Bind it from the root profiles.yaml, --profiles, or --agent (solver or *). Lock fails with missing_binding when none of those supply it. --oracle runs solution/solve.sh in the environment phase and does not send instruction.md to the agent.
Common ageval run flags
| Flag | Meaning |
|---|---|
--task <id> | One member; omit = run the whole dataset. Repeatable with --layout harbor |
--layout harbor | Read an unmodified Harbor task directory or a parent of task directories. Omit keeps the ageval package layout. A single Harbor task rejects --task. registry.json indexes are rejected |
--oracle | With --layout harbor, run solution/solve.sh and skip the agent. Omit leaves solution/ unused |
--max-concurrent-tasks N | Max concurrent units; speeds wall time only |
--n-attempts / -k N | Always-k: N independent Attempts per task (default 1); CLI/job only |
--resume <attempt_id> | Continue that unsealed Attempt in place. Requires one --task. Exit 2, and no new result.json, when result.json already exists, the lock digest does not match, or the environment is still running the Attempt |
--resume-suite <suite_run_id> | Continue an existing suite. Skip slots that already settled as PASS, FAIL, or ERROR. An unfinished slot whose activity log matches this suite, task, and attempt index continues in place. A slot with no matching log starts a new Attempt. Recompute pass@k / pass^k |
--retry-errors | With --resume-suite: re-run finished ERROR rows in this filter. PASS / FAIL stay skipped. New Attempt; old row stays in previous[] |
--replace-slot | With --resume-suite --task: re-run that finished slot; new Attempt; old row stays in previous[] |
--attempt-index N | Always-k slot to replace (default 0). Only with --replace-slot |
--keep-workspace | Keep the environment work root after cleanup (default: delete). Uploads never pack it. Docker volumes are still removed. Does not keep activity.jsonl or checkpoints/ |
--keep-recovery-state / --no-keep-recovery-state | After result.json and trajectory.jsonl exist, keep or delete activity.jsonl, checkpoints/, and replay-input.json. Default: delete. An interrupted Attempt always keeps them, so --resume does not need this flag. Unset falls back to AGEVAL_KEEP_RECOVERY_STATE. Independent of --keep-workspace |
--keep-vendor-raw | After seal, keep vendor raw and intermediate invocation event files. Default: drop. task-artifacts/ stays either way. Independent of --keep-workspace |
--dir <path> | Hub ref only. Looks at <path>/<dataset_id>/; reuses a matching child |
--agent | Bind an ageval.agent/1 Agent package (pi, org/name@version, or a local path); mutually exclusive with --profiles |
--fork <attempt_id> | Start a new Attempt from a retained checkpoint of that source. Requires --checkpoint and --model. Mutually exclusive with --resume. Rejects --agent and --profiles |
ageval replay <dataset> --run <attempt_id> re-executes retained terminal actions in a new environment. It does not call the solver model. --evaluate grades the new artifacts. Without it, completion is not PASS or FAIL. The source Attempt is unchanged. Default pass@k skips the child.
| --checkpoint <seq> | Activity seq to fork. The cut must be a completed run-phase snapshot before solver sealing |
| --model | Override model.id on roles --agent bound. Requires --agent, except with --fork, where it sets the continuation model and does not replace the harness. Not results upload-suite --model. results analyze --model is a separate flag and still requires --agent |
| --set / --profiles | Overrides / alternate job binding |
| --probe | Feasibility only: plan + readiness for this binding/environment. No Agent, no image write, no lock digest change. Exit 1 if the selected path is unsatisfied. |
| --json | Write the full result document on stdout. A TTY defaults to a short recap (suite: summary path plus ageval view / upload-suite); a pipe still defaults to JSON. |
status / cancel
| Command | Meaning |
|---|---|
ageval status <run_id|suite_run_id> | ControlStore; add --dataset for suite progress.json (system suite id defaults to 8 hex) |
ageval cancel <run_id|suite_…> | One Run or suite job; suite may use --dataset → cancel.requested |
--set
| Pointer | Meaning |
|---|---|
/parameters/seed | Seed etc. |
/parameters/harbor_oracle | With --layout harbor, same switch as --oracle |
/parameters/active_profile | Profile id |
/agent_profiles/<role>/model | Nested model object |
/agent_profiles/<role>/model/id | Model id |
/agent_profiles/<role>/model/api | chat or responses |
/agent_profiles/<role>/model/timeout | Per-request HTTP timeout |
/agent_profiles/<role>/model/watchdog | Abort clock |
/agent_profiles/<role>/model/reasoning/effort | Thinking effort. Omit means do not send. max is the pin's highest tier |
/agent_profiles/<role>/executor | Executor id |
/agent_profiles/<role>/api_key | Env locator name |
/agent_profiles/<role>/base_url | Literal URL or env locator |
/agent_profiles/<role>/options/<key> | Job binding override (entry, plugin agent, …; ACP still rejects command / engine keys) |
Intent limits.* | Not overridable via --set |
Anything else fails at lock. Details: src/ageval/cli/README.md · skill ageval-cli.