Environments
Host, Docker, e2b, ssh, daytona. ACP does not change.
Set the environment in profiles.yaml with environment:. ACP only needs the environment to provide attach_stdio. Do not write plugin_id: docker. Changing the environment does not change ACP. exec is a one-shot command on the same environment, not a second environment. On docker, startup writes the bound options.entry into the image; if that entry is already there, it does not run npm i. Switching model does not rebuild; switching entry does.
| Environment | Opens | To actually run |
|---|---|---|
local | Host directory | Default examples/datasets/tau3-airline-5 |
docker | Local container | Default examples/datasets/minimal-demo profiles.yaml |
e2b | Cloud sandbox | E2B_API_KEY; extra: uv tool install 'ageval-cli[e2b]'; missing extra/key → --probe ready: false |
ssh A | Remote machine | AGEVAL_SSH_HOST / USER / KEY |
ssh B | Remote existing image | Same locators plus image: already on the host |
daytona | Cloud sandbox (OCI snapshot) | DAYTONA_API_KEY; missing key → --probe ready: false. Default CI does not run live Daytona |
ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg
ageval run examples/datasets/tau3-airline-5 --task airline-00
ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg --probeMissing extras or keys: --probe must report ready: false and started: false. ACP only sees attach_stdio.
ssh A does not support live ACP stdio. dsh / nooa / openhands on that surface use host.exec, not a long ACP session.
environment/setup.sh is the last step of the environment phase, not its own phase. Gold (evaluation/) enters the scoring host at evaluate. profiles.yaml may set evaluate_host.isolated: true (Current: docker) to start a second scoring environment so gold never enters the Agent environment; omit it to stay on the run environment.
Several scoring images (different installed closures, sequential stages) are declared on the member as evaluation.environments: each name has a dockerfile and/or docker_image under environment/evaluate/<name>/, never under the gold tree. Core starts a host on first scoring.exec / session(environment=); unused names are not built. Setting the map requires evaluate_host.isolated + docker and ignores environment/evaluate.Dockerfile. ACP attaches to the named host; HTTP judges ignore environment=. --agent and --profiles are mutually exclusive. --model requires --agent.