Task authoring tutorial

Copy a task from examples/datasets/minimal-demo, edit run.py and the evaluator, run it.

Goal: lock, run, and get an evaluation result on this machine.

Prerequisites

Tutorial commands use in-repo examples/… paths, so clone the repo first. The CLI itself installs with uv tool install ageval-cli; uv sync is only for changing ageval itself.

Have an agent write it

npx skills add ZJU-REAL/ageval --skill ageval-config-package
npx skills add ZJU-REAL/ageval --skill ageval-sdk-harness

Describe the task you want and your coding agent produces the directory, task.yaml, run.py, and evaluator.py. Copying the steps below by hand works just as well.

Copy a tasks/<id>/ directory from examples/datasets/minimal-demo (start from terminal-jsonl-agg). The directory name must equal task_id.

format: ageval.task/1
task_id: hello
parameters:
  instruction: "Reply DONE"
agent_profiles:
  - id: solver
limits:
  wall_time_seconds: 300
  environment_seconds: 600
  evaluate_seconds: 600
  agent_invocations: 1
artifacts:
  publishable:
    - id: reply
      path: artifacts/reply.json
evaluation:
  inputs:
    - artifact: reply
      target: artifacts/reply.json

Do not write harness:, provider:, or assurance. The environment and ACP entry live in profiles.yaml. wall_time_seconds covers the run only. environment_seconds and evaluate_seconds default to 600. agent_invocations counts run-phase invokes.

ageval lock examples/datasets/minimal-demo --task hello
ageval run  examples/datasets/minimal-demo --task hello

Scoring defaults to a script against gold. For LLM-as-judge, add a role on the same profiles.yaml and Agent.session after gold lands. For several scoring images with different closures scored in sequence, declare evaluation.environments on the task; the evaluator names the host for each stage. Optional checks on the evaluator return (helper evaluation_check) become evaluation/checks.json for Verifier.