Swap Agent or environment with plugins

Swapping the Agent or the environment does not rewrite the dataset: install the plugin and switch it in profiles.yaml.

ACP is the default path (pi / Codex / Claude Code / OpenCode). dsh, nooa, miniswe, terminus-2, and openhands are the same kind of plugin. Switching environment: docker to e2b, ssh, or daytona changes that one line in the ACP profiles — not ACP code.

Install

ageval plugin install plugins/nooa
# or: ageval plugin install plugins/dsh
# or: ageval plugin install plugins/miniswe   # model on the host; bash via environment.exec
# or: ageval plugin install plugins/terminus-2  # model on the host; tmux via environment.exec
# or: ageval plugin install plugins/openhands  # pinned SDK inside the attempt environment
# or: ageval plugin install plugins/acp-oneshot  # one-shot ACP pair in the environment; exec only
ageval plugin list

Install updates $AGEVAL_HOME/plugins (default ~/.ageval/plugins) only. It never rewrites profiles.yaml, ageval.yaml, or task.yaml.

plugin.yaml may include description: one paragraph Hub shows above Install (CLI). Markdown links ([text](https://…)) render there.

Uninstall:

ageval plugin uninstall nooa

Switch in config

format: ageval.profiles/1
environment: local
agent_profiles:
  solver:
    executor: nooa
    extensions:
      - plugin: nooa
        options:
          agent: "lib.agents:JsonlAggAgent"
          method: "run"

executor: selects the Agent for this profiles file. Plugin options live on that plugin's extensions row. Docker image layers are written when the environment starts. ACP entry lives on - plugin: acp / options.entry and is written into the image; switching model does not rebuild.

ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg \
  --profiles examples/datasets/minimal-demo/profiles.nooa.yaml

ageval lock writes the resolved graph as extension_bindings. If capabilities or credentials do not match, the check fails and the run does not start.

ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg \
  --profiles examples/datasets/minimal-demo/profiles.dsh.yaml --probe

--probe checks whether this environment can open here: host_requires, Docker daemon (docker), locator names. If something is missing, the command exits non-zero and the run does not start.

Installed is not the same as runnable

ageval executors.host_ready is not “installed” and not “layers are in the image”. Installed plugins that are not listed in extensions stay off this run's image.

Relation to ACP

The default coding-agent path remains executor: acp + options.entry (parent JSON-RPC, needs attach_stdio). acp-oneshot is the other path: a one-shot client+server inside the environment, over exec only. Plugins are optional; the same dataset switches only via profiles. miniswe keeps the model on the host and sends bash through environment.exec. terminus-2 does the same with a tmux pane: the plugin installs tmux in the attempt environment when it is missing. Missing exec fails at lock. openhands runs the pinned SDK on the attempt workspace (exec, upload, download). The instruction is a file in that environment.

PASS still comes only from evaluator.py.

npx skills add ZJU-REAL/ageval --skill ageval-plugin