ACP profiles
Plug coding agents with executor: acp and - plugin: acp options.entry.
Recommended config
# profiles.yaml
format: ageval.profiles/1
environment: local
agent_profiles:
main:
executor: acp
extensions:
- plugin: acp
options:
entry: codex
# idle_timeout_seconds: 120
model:
id: gpt-5.4-mini
# reasoning:
# effort: high # optional; must be a value the session advertises| Field | Note |
|---|---|
executor | Use acp for coding agents |
- plugin: acp / options.entry | Required; discover with ageval executors -v |
model.reasoning.effort | Optional. Entries with an advertised thinking selector use set_config_option. grok-build adds --reasoning-effort on grok agent … stdio. Omit to leave the entry default |
options.idle_timeout_seconds | Optional stall ceiling during session/prompt. Any session/update or permission resets it. Unset / ≤0 → the phase clock only. Fires acp_idle_timeout (still capped by that clock: wall_time_seconds in run, evaluate_seconds in evaluate) |
model.id | Model id for that entry |
api_key | Optional env name |
No command/version/install in package yaml.
Evaluate-phase ACP (for example judge) must Agent.session("judge", environment="<name>") when the task declares evaluation.environments, the named scoring-host map; attach targets that scoring host. Naming a host during run, or omitting the name when the map is set, fails once. HTTP judges ignore environment=.
What's installed on this machine
ageval executors -vageval executors -v reports three signals per entry:
| Signal | Meaning |
|---|---|
engine_ready | the entry's agent runtime (the pi / codex CLIs) is installed on this host |
acp_entry_ready | the ACP entry on top of that runtime is ready |
credential_missing | every declared credential env name is unset and the binding has no api_key |
Entries that require a key (for example pi) fail --probe / session-open, and the run does not start; entries that document keyless (OAuth) auth warn only. The child environment is allowlisted (PATH / HOME / LANG, the entry’s credential names, and the binding’s api_key / base_url). Undeclared host tokens do not reach the entry.
Runtime switch
ageval run <dataset> --task <id> \
--set '/parameters/active_profile="main"'One ACP client in the parent process; each vendor's protocol translation runs outside the process, in the entry. --agent pi --model glm-4.7 picks this run's agent and model directly; non-ACP agent runtimes take a whole --profiles file instead.