Run a single task

Command

ageval run <dataset> --task <task_id>

Example:

ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg

# Always-k: fixed k independent Attempts for one task (CLI only; default k=1)
ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg -k 5

Config only, no agent:

ageval lock examples/datasets/minimal-demo --task terminal-jsonl-agg

--task must match the member directory name and task_id in task.yaml.
On a TTY, stdout is a short recap (single task: logs path). A pipe or --json still writes the JSON document. With k>1, the run is a suite job under .ageval/suite-runs/.

Resume an interrupted Attempt

ageval run <dataset> --task <task_id> --resume <attempt_id>

Use this when the parent process stopped before result.json was written. ageval keeps that Attempt id. Completed steps stay in .ageval/runs/<attempt_id>/activity.jsonl. A live environment that still matches the last completed step is reused. A lost environment is started again and restored from the latest checkpoint of /attempt/workspace and /attempt/home. Completed calls are not run again. Wall-clock time and invocation count already spent stay charged.

The command exits 2 and does not write result.json when that file already exists, the locked configuration digest does not match, or the environment is still running this Attempt.

Fork a retained checkpoint

ageval run <dataset> --task <task_id> --keep-recovery-state
ageval checkpoints <dataset> --run <attempt_id>
ageval run <dataset> --task <task_id> --fork <attempt_id> --checkpoint <seq> --model <model_id>

--keep-recovery-state keeps activity.jsonl and checkpoints/ after the source has sealed. ageval checkpoints lists the cuts. --fork starts a new Attempt from one completed run-phase cut, on a fresh environment, with the source harness and the model you pass. The parent Attempt stays sealed. Default pass@k skips the child.

Replay recorded terminal actions

ageval run <dataset> --task <task_id> --keep-recovery-state
ageval replay <dataset> --run <attempt_id>
ageval replay <dataset> --run <attempt_id> --evaluate

ageval replay starts a new Attempt and a fresh environment from the task's initial setup. It re-executes the retained terminal actions in order and does not call the solver model. The report names the source Attempt, the replay status, the first divergence, and the new evidence path. --evaluate grades the new artifacts with evaluator.py. Without it, replay completion is not PASS or FAIL. Default pass@k skips the child. The source Attempt is left unchanged.

What you get

• A Result summary on the terminal
• A logs path under the Dataset root: .ageval/runs/<run_id>/ (when k=1)
• For k>1: job metrics such as pass@k / pass^k in suite summary.json

PASS / FAIL comes only from evaluator.py. A low score is FAIL. When the run hits wall_time_seconds or agent_invocations, the evaluator still scores and result.json records that key in limit. An environment or evaluate clock expiry, or an evaluator status other than PASS or FAIL, is ERROR.

Common failures