Config field reference

Field rules:

Root ageval.yaml

Not in yaml: n_attempts / Always-k (CLI/job only).

A dataset with no tasks fails lock. Unknown format → invalid_format.

Member task.yaml

Optional evaluation.* fields:

Files present are recognized: run.py, evaluator.py, Dockerfile, setup.sh.

Job binding (profiles.yaml)

format: ageval.profiles/1
environment: docker
agent_profiles:
  solver:
    executor: acp
    api_key: ${ZHIPU_API_KEY}
    model:
      id: gpt-5.4-mini
    options:
      entry: pi
    extensions:
      - plugin: acp
      - plugin: docker

Optional label on a role. --agent is mutually exclusive with --profiles. --model requires --agent and writes this run’s model.id. Optional overlays paths must start with overlays/ and stay locator-only.

model is a nested object. Omitting it is still valid. A string model, or a flat model key on options (api, reasoning_effort, timeout, request_timeout, watchdog, extra_body), fails lock. Optional leaves are api (chat or responses), reasoning.effort, timeout, watchdog, and extra. An omitted leaf is not sent. reasoning.effort: max is an alias: lock rewrites it to that model's highest pin effort tier. A model the pin cannot join, or a pin row with no effort list, fails lock. options.upstream_stall_seconds defaults to 3600; 0 turns that wait off. The retry interval is 60 seconds and is not a field.

Optional evaluate_host.isolated: true (Current: docker) starts a scoring environment. Combined with member evaluation.environments, Core lazily starts one host per name.

Task parameters cpus, memory_mb, gpus, gpu_types, and verifier_gpus are docker container limits when set. A required GPU the host cannot show fails the attempt with environment_gpu_unavailable. Omitted limits leave today's docker run shape.

Optional docker environment_options: image / platform / network / user / egress / egress_allow / python_version. python_version is the official image CPython minor (e.g. "3.13"; omit = 3.12, shape ^\d+\.\d+$); an image that cannot be pulled fails once, with no fallback. egress: llm allows outbound traffic only to bound profile base_url hosts; egress_allow on the same map is extra hostnames (omit = none). evaluate_host.environment_options may set network / egress / egress_allow / platform / user / python_version (not image); when omitted, the scoring host inherits the job's platform / user / python_version and does not inherit the agent's egress / egress_allow / network.

A task environment/compose.yaml starts its services with docker compose up -d. The agent container is still its own docker run. A depends_on entry that names main is omitted for that up. Healthchecks on those services are not waited on. On the compose project network the agent container is also registered as hostname main, so a sidecar can dial the agent by that name. network_mode: service:<name> and a run with no compose file do not add that alias. network_mode: service:<name> — from environment/docker-compose.yaml service main, task parameters.network_mode, or environment_options.network_mode — runs the agent with --network container:<that service's container>. The agent and that service share one network, so 127.0.0.1 is that service's 127.0.0.1. cap_add and main environment (parameters.cap_add, parameters.main_environment, or the same environment_options keys) are set on that docker run. The scoring container does not use them. The values stay on the docker run and are not copied into the lock or evidence.

Allowed top-level dirs

prompts · schemas · environment · evaluation · data · lib · upstream · solution

Common lock failures

id mismatch · bad format · illegal dataset_id · docker without Dockerfile · missing ACP entry · secrets in yaml · illegal folders · no tasks