Results and PASS

Do not mix these up

1. run.py completed ≠ PASS
2. Trajectory / tokens / agent self-report ≠ score (including evaluation/observation.jsonl and evaluation/checks.json)
3. PASS / FAIL comes only from the evaluator.py return value
4. Whole-dataset / job aggregates (including pass@k) ≠ a PASS for the whole dataset
5. A post-run analyzer report ≠ a score, and it does not rewrite evaluation/checks.json

On the terminal

• Result: status and evaluation for this run
• logs: evidence path under the dataset root (e.g. .ageval/runs/<run_id>; typical when k=1)
• Whole-dataset pass_rate / mean_score: job summary

Always-k and pass@k / pass^k

After Always-k (-k / --n-attempts), the whole-dataset summary.json → metrics includes:

Attempts that carry fork stay out of the default pass@k and pass^k counts. They remain visible as their own rows.

Contract: metrics.pass_at_k / pass_power_k

Each key is a stringified k ("1", "2", "5", …). Value shape:

{
  "1": { "value": 0.75, "n_tasks": 4, "incomplete_tasks": 0 },
  "2": { "value": 0.5,  "n_tasks": 3, "incomplete_tasks": 1 }
}

Upload / Registry:

• ageval results upload-suite ensures / recomputes k metrics locally (from attempts[] or per-task n/c) before POST of the full metrics blob.
• Registry stores metrics_json as-is; list/get suite does not strip pass_at_k / pass_power_k.
• First-class columns remain pass_rate / mean_score; k metrics live under metrics.
• Not part of config_fingerprint / job identity; different k jobs may sit side by side.
• Legacy suites without k data: missing metrics.pass_at_k → Hub shows —.

task_refs (multi-attempt): may include n, c, attempt_run_ids (plus primary run_id) for audit and --with-attempts.

What Hub actually lists

So: to show whole-dataset results on the public board, run the whole dataset → upload-suite against a published release. Uploading one task is not enough.

Local view

ageval view opens a dataset root, a directory of dataset roots, or a cached registry ref. The header is Datasets, Jobs, Agents, and Plugins. Jobs → Tasks → Attempt is the results path for the package you pick.
Attempt pages may show Timing (and Tokens when present). When analysis/<analyzer>/report.json exists, the evidence tabs include Analysis. It lists those observations and is not a score. Trajectory steps follow file order (thought next to the tools it precedes). A step may show a duration when the row has elapsed_ms. Verifier may show evaluation/checks.json as a structured preview in the file tree (and a Trajectory / Files toggle when observation.jsonl is also present).

Failure kinds

Exit codes: current CLI + skill ageval-cli.