Switch agent and model
Swap coding-agent entries with profiles, --agent, --model, or --set.
One task can declare several agent_profiles and pick which to use at run time.
What’s on this host?
ageval executors -vCheck engine_ready / acp_entry_ready per ACP entry.
Declare in config
# profiles.yaml
format: ageval.profiles/1
environment: docker
agent_profiles:
solver:
executor: acp
extensions:
- plugin: acp
options:
entry: pi
model:
id: gpt-5.4-miniACP entry lives on executor: acp and options.entry. Packaged ids are the built-ins below. $AGEVAL_HOME/acp-entries.json (default ~/.ageval/acp-entries.json) can add an id; it cannot replace a packaged id. The process that starts is that entry's pinned package binary. The official Attempt image does not include entries from that file. --agent still resolves only the packaged ids.
Agent packages (--agent)
ageval.agent/1 is an Agent package (executor, entry, overlays); model.id is the default when --model is omitted, not identity.
| Goal | How |
|---|---|
| Built-in package | --agent pi. Built-ins pi / opencode / codex / claude-code / grok-build / openai-http / anthropic-http ship with ageval — no install |
| Custom overlay pack | ageval agent install org/name@0.1.0 first, then bind. Install also pulls binding.extensions[].plugin (skips contrib; a missing plugin or unsatisfied host_requires rejects the whole install) |
| Model for this run only | --model <name> (with --agent; omit to keep the package default. Not ageval results upload-suite --model — that is an observational upload label) |
--agent and --profiles are mutually exclusive.
ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg --agent pi --model glm-4.7
ageval agent install examples/agents/dsh-default
ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg --agent local/dsh-default@0.1.0Temporary switch
ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg \
--set '/agent_profiles/solver/options/entry="codex"'Only allowlisted --set pointers work, e.g. /agent_profiles/solver/model/reasoning/effort="high". The CLI page lists the pointers.
api_key is always an env name, never the secret. Optional profile fields: model.reasoning.effort (thinking level), options.idle_timeout_seconds (stall ceiling), and base_url.
Non-ACP agent runtimes (nooa / dsh / miniswe / terminus-2 / openhands) switch with a whole --profiles file. They ship as plugins: ageval plugin install plugins/<name>, then set executor: to that name and attach the matching plugin: line in profiles. openhands runs the pinned SDK on the attempt workspace and reads the task instruction from a file there.