Package layout

npx skills add ZJU-REAL/ageval --skill ageval-config-package

Two layers

CLI path is always the Dataset root; --task selects a member.

my-dataset/
├── ageval.yaml
├── profiles.yaml           # optional job binding defaults
├── shared/                 # optional cross-task share (in packageDigest)
│   ├── lib/                # import as shared.lib.*
│   ├── assets/             # read-only domain data (code paths)
│   └── README.md
└── tasks/
    └── hello/
        ├── task.yaml
        ├── run.py
        ├── evaluator.py
        ├── prompts/
        ├── environment/    # Agent Dockerfile; scoring recipes in evaluate.Dockerfile or evaluate/<name>/
        ├── evaluation/     # gold — not for Agent by default; no scoring Dockerfiles
        ├── data/
        └── lib/            # task-only; import as lib.*

Allowed task top-level dirs: prompts, schemas, environment, evaluation, data, lib, upstream, solution.

Optional Dataset-level shared/

Homogeneous multi-task packages can keep shared bridges and domain assets under root shared/ instead of copying into every tasks/*/lib/.

The Core-injected path prefix is [task_dir, dataset_root] — the shared/lib leaf is not injected as a path root.

Hard rules (summary):

• If present, shared/** must enter packageDigest / the publish blob
• Not Agent-mounted by default; gold stays under tasks/*/evaluation/ only
• Task members must not own top-level name shared (shared/ dir or shared.py) — lock fails
• Same stem under shared/lib and tasks/*/lib is allowed (namespaces differ)
• Core never silently COPYs shared/ into docker images — task Dockerfile must declare it and put dataset root on PYTHONPATH
• Author gate: uv run python scripts/check_shared_lib_collisions.py <dataset-root>

Having shared/, changing the import contract, or Hub Shared browse does not make the evaluation results more credible.

Root ageval.yaml

format: ageval.dataset/1
dataset_id: org/my-suite
version: "0.1.0"
tasks:
  root: tasks
# Optional dataset-level preparation (runs before any Attempt opens):
# assets:
#   - path: tasks/clip-01/data/clip.mp4
#     url: https://cdn.example.com/clip-01.mp4
#     sha256: <64 hex>
# prepare: ./prepare.sh   # gated: needs --allow-prepare or a trusted dataset

Large files that a script can fetch belong in assets, not in git. ageval prepare (also run automatically by run / lock) downloads each missing asset, verifies its sha256, and places it at path; a mismatch stops the run before any Attempt opens. A prepare script is disabled by default and runs only with --allow-prepare or for trusted datasets; declared assets are re-verified after it runs.

dataset_id has a strict charset; it is not part of scoring.
Do not put harness / provider / agent_profiles / evaluation on the root.

parameters vs envelope

• parameters → run.py (ctx.params)
• limits / evaluation / environment/ → Core
• Environment and ACP entry → root profiles.yaml

No second config truth in run.py. Allowed runtime overrides go through --set.