Write an evaluator

The evaluator owns scoring and truth — no agent start, no host secrets. PASS/FAIL comes only from here (then the Result).

npx skills add ZJU-REAL/ageval --skill ageval-config-package
npx skills add ZJU-REAL/ageval --skill ageval-sdk-harness

Config snippet

artifacts:
  publishable:
    - id: session-output
      producer: run
      path: artifacts/session-output.json
      media_type: application/json

evaluation:
  runtime: python
  entrypoint: evaluator:evaluate
  network: none
  # tmpfs_mb: 256   # optional in-environment evaluator /tmp (MiB); omit → 32. Not limits.*
  inputs:
    - artifact: session-output
      target: artifacts/session-output.json
  output:
    format: json

evaluator.py is a parent subprocess (same shape as run.py). A profiles file may set evaluate_host.isolated: true (Current: docker) to start a scoring environment so gold never enters the Agent environment. Sequential stages with different installed closures use evaluation.environments on the task; the evaluator names a host with inputs["scoring"].exec(<name>, argv) or Agent.session(..., environment=<name>). Unused names are not started. Omit the map → the single scoring host (or same-environment evaluate); scoring.exec hits that default host. A named map still requires an exact name; unknown names fail once and do not start.

Two scoring styles

Default is a deterministic script: read published artifacts and gold under evaluation/, return {status, score, metrics}; status is PASS or FAIL. A low score is FAIL. Any other status, including ERROR, makes the attempt ERROR. A run-phase limit is still this return value, and result.json limit names the key. A dead environment or a crashed evaluator is ERROR. No Agent call. No evaluation/observation.jsonl. Optional extra key checks (helper evaluation_check) is written as evaluation/checks.json for Verifier; bind ignores it. Without checks, Verifier stays result.json plus the file tree.

Optional LLM-as-judge: add another role on the same job profiles.yaml (e.g. judge) and list that id on task.yaml agent_profiles. After gold is uploaded, evaluator.py may Agent.session("judge").invoke(...). The dataset owns the prompt. invoke must not override profile_id / executor.

from ageval_sdk import evaluation_check

def evaluate(inputs):
    # inputs["agent"] is injected on this machine (parent socket); omit → script path
    ...
    return {
        "status": "PASS",
        "score": 1.0,
        "metrics": {},
        "checks": [
            evaluation_check(
                "files",
                title="artifact files",
                status="PASS",
                script="evaluator.py",
            ),
        ],
    }

checks is optional. Omit it and parent does not write evaluation/checks.json. A check row may be FAIL while the return is PASS — bind still reads only status / score / metrics. script is a dataset path (member-relative, or tasks/<id>/…); Verifier opens it from the package, and does not copy the script into the Attempt archive.

Named scoring hosts

When sequential stages need different installed closures, do not reuse environment/evaluate.Dockerfile. On the member yaml:

evaluation:
  environments:
    audit:
      docker_image: python:3.12-slim          # existing OCI tag
    verification:
      dockerfile: environment/evaluate/verification/Dockerfile

Recipe files live under environment/evaluate/<name>/, never under evaluation/ (that tree is gold). Setting the map ignores evaluate.Dockerfile / evaluation.docker_image. The job must set evaluate_host.isolated: true and environment: docker.

from ageval_sdk import evaluation_check

async def evaluate(inputs):
    scoring = inputs["scoring"]
    audit = await scoring.exec("audit", ["python", "/attempt/evaluation/audit.py"])
    if audit.exit_code != 0:
        return {
            "status": "FAIL",
            "score": 0.0,
            "metrics": {"stage": "audit"},
            "checks": [
                evaluation_check(
                    "audit",
                    status="FAIL",
                    script="evaluation/audit.py",
                    environment="audit",
                    exec_result=audit,
                ),
            ],
        }
    async with inputs["agent"].session("judge", environment="verification") as session:
        reply = await session.invoke(inputs["parameters"]["verify_prompt"])
    ok = bool(reply.get("ok"))
    return {
        "status": "PASS" if ok else "FAIL",
        "score": 1.0 if ok else 0.0,
        "checks": [
            evaluation_check(
                "audit",
                status="PASS",
                script="evaluation/audit.py",
                environment="audit",
                exec_result=audit,
            ),
        ],
    }

exec argv is a list; stdout / stderr / exit_code come from inside that container. The first use of a name starts that host and uploads the snapshot + gold. Unknown names, run-phase environment=, or omitting the name on a named map fail once and do not start. v1 is sequential stages only — do not run exec and session concurrently on the same name.

No Agent.session = the script path. Verifier shows the evaluation file tree; checks.json renders as a structured preview in that tree. With an evaluate-phase observation.jsonl as well, pick Trajectory or Files from the toggle (trajectory first). result.json stays in the file tree. Browse locally with ageval view.