Authoring
Write run.py
The task loop lives in the dataset; call the agent through the SDK.
run.py owns the loop, roles, local tools, and published artifacts.
| You own | Do not |
|---|---|
| loop, roles, tools | PASS, host credentials, isolation |
SDK session / invoke | A vendor SDK as Core |
Scoring lives in evaluator.py: a deterministic script against gold, or an optional LLM-as-judge after gold lands.
Have an agent write it
npx skills add ZJU-REAL/ageval --skill ageval-sdk-harnessDescribe the loop you need and let your coding agent produce run.py; the snippet below is what it must satisfy.
from ageval_sdk import RunContext, RunTerminal
async def run(ctx: RunContext) -> RunTerminal:
async with ctx.agent.session("solver", max_turns=1) as session:
reply = await session.invoke(ctx.params["instruction"])
ctx.publish_json("reply", {"ok": bool(reply.get("ok")), "text": reply.get("text") or ""})
if not reply.get("ok"):
return RunTerminal.failed(str(reply.get("error") or "invoke_failed"))
return RunTerminal.completed("ok")session.invoke may take tools and messages. openai-http /
anthropic-http return structured tool_calls; ACP ignores the catalog
and stays on text invoke. Omit both for prompt-only.
Files present are recognized. Do not declare a harness entrypoint.