Run the whole dataset

Omit --task and ageval run schedules every member — each locks, runs, and evaluates independently:

ageval run examples/datasets/minimal-demo
ageval run examples/datasets/minimal-demo --max-concurrent-tasks 2

Two axes (do not mix)

# Full suite, k=5 per task (CLI/job only)
ageval run examples/datasets/minimal-demo -k 5 --max-concurrent-tasks 2

# Always-k on one task
ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg -k 5

# Continue an existing suite: recover unfinished slots in place, skip settled PASS / FAIL / ERROR
ageval run examples/datasets/minimal-demo --resume-suite <suite_run_id> --task terminal-jsonl-agg -k 5

# Replace one finished row (PASS / FAIL / ERROR): new Attempt, same suite id
ageval run examples/datasets/minimal-demo --resume-suite <suite_run_id> --task terminal-jsonl-agg --replace-slot
# Always-k: one (task_id, attempt_index)
# ageval run examples/datasets/minimal-demo --resume-suite <id> --task terminal-jsonl-agg --replace-slot --attempt-index 1

# Retry every finished ERROR in this filter (PASS / FAIL stay skipped)
ageval run examples/datasets/minimal-demo --resume-suite <suite_run_id> --retry-errors

# Regrade one finished Attempt from its harvested artifacts (new Attempt; the agent does not run)
ageval results regrade examples/datasets/minimal-demo --run <attempt_id>

Hard rules:

• n_attempts is not allowed in task.yaml and is not part of config_fingerprint
• Resume without --replace-slot or --retry-errors skips settled PASS / FAIL / ERROR. An unfinished slot whose activity.jsonl header matches this suite, task, attempt index, and lock digest continues in place. A slot with no matching header, or a digest mismatch, starts a new Attempt. Missing Always-k indices are still appended
• --retry-errors (with --resume-suite) re-runs finished ERROR rows in this filter. PASS and FAIL stay skipped. Each retry is a new Attempt; the old row stays in previous[]
• --replace-slot writes a new Attempt; the old row stays on disk and in previous[]; metrics use the new current only
• ageval results regrade mints a new Attempt, copies the source Attempt’s task-artifacts/ onto the scoring host, and runs evaluate only. Gold and evaluator.py come from the dataset. The source result.json stays. A missing harvest is ERROR. Trajectory text is not a verdict. There is no task field that turns regrade on or off
• task-artifacts/ stays after the default slim. Cleaning up the live workspace does not remove that snapshot
• Hub / Viewer history is patch N plus that Attempt’s start time (not the suite replace clock, not the run_id)
• Rows that only have a suite-cancel placeholder (never actually ran) are retried on resume so pass@k is not permanently deflated
• Progress / cancel below; Hub Leaderboard shows pass@k / pass^k / n_attempts when metrics.pass_at_k is present; local metrics are always written after Always-k
• Default system suite_run_id is 8 hex (no suite_ prefix)

Aggregate metrics

Under .ageval/suite-runs/<suite_run_id>/summary.json → metrics:

Tasks with too few samples are omitted from that k’s denominator.

Same directory: progress.json. Attempt result.json may include phase_timing (environment / run / evaluate / record / cleanup).

Progress and cancel

ageval status <suite_run_id> --dataset examples/datasets/minimal-demo
ageval cancel <suite_run_id> --dataset examples/datasets/minimal-demo

• status — ControlStore record + suite progress when present
• cancel — writes cancel.requested (no new units); SIGTERM stored pid if any

Running the whole dataset is the task axis; campaign sweeps parameters on one task. Always-k is not the campaign matrix axis.

Results land under .ageval/suite-runs/. Open the dataset in the local viewer to see Timing bars. With Kind set to suite, those jobs can also switch to a waffle: one square is one trial.

Upload to Hub (Leaderboard / Jobs)

Hub Dataset → Leaderboard and Task → Jobs list whole-dataset results in Registry, not a feed of single-task uploads.

ageval run examples/datasets/minimal-demo --profiles examples/datasets/minimal-demo/profiles.yaml
# suite_run_id e.g. d3e5fba6
ageval results upload-suite examples/datasets/minimal-demo \
  --suite-run d3e5fba6 --public --with-attempts \
  --agent acp-pi --model zai-coding-cn/glm-5.2

Common mistake: only ageval run … --task X then ageval results upload --run … — the attempt may exist in Registry, but Leaderboard usually still has no row. Leaderboard needs a suite job + upload-suite.

The public board has two more gates: the suite must be complete (every task on the bound version has a result; FAIL / ERROR still count) and bound to a release. Incomplete or draft-bound rows stay on Task Jobs. Upload a working tree with ageval publish --draft, then an owner runs ageval release — or ageval publish a release directly.