Task anatomy · field dictionary
Every required file — and what each field means
Use this as your cheat sheet for structure. If someone asks “what does number_of_milestones mean?” — it’s here. Learn the meanings; don’t memorize an answer key.
Folder structure
my-task/ (or tasks/<track>/<slug>/)
├── task.toml ✅ REQUIRED — metadata + timeouts + resources
├── instruction.md ✅ REQUIRED — only prose the agent reads
├── environment/ ✅ REQUIRED — agent-visible world
│ ├── Dockerfile ✅ REQUIRED (single-container) — digest-pinned base
│ ├── .dockerignore ✅ REQUIRED — must exclude solution/ and tests/
│ ├── docker-compose.yaml ⚪ optional — multi-container only (+ flags in toml)
│ └── app/, data/, docs… ⚪ optional — starter code / specs the agent may see
├── solution/
│ └── solve.sh ✅ REQUIRED — oracle; must earn reward 1 every run
└── tests/
├── test.sh ✅ REQUIRED — ALWAYS the verifier entrypoint
└── test_outputs.py ✅ REQUIRED in practice — pytest assertions
Special paths at runtime (Harbor mounts these):
/logs/verifier/ ← reward.txt lives here
/logs/agent/ ← agent logs
/oracle/ ← solution copied here when oracle runs
/tests/ ← tests copied here at verify time (not in agent image)Always required
- task.toml, instruction.md
- environment/Dockerfile + .dockerignore
- solution/solve.sh
- tests/test.sh (+ pytest checks)
Do not put in the zip parent
- Extra README / jobs/ logs / unrelated data/
- difficulty.md / explanation.md (explanations live on Snorkel)
- rubrics.txt covering everything offline — use the UI generator
- Ground-truth expected_output.json inside environment/
task.toml — every field explained
Edition 2 uses version = "2.0". Required sections: [metadata], [verifier], [agent], [environment]. Do not add a top-level [task] table for this project’s policy.
Annotated example (Edition 2 shape)
version = "2.0" [metadata] author_name = "anonymous" # who wrote it (anonymous OK) author_email = "anonymous" # contact (anonymous OK) difficulty = "unknown" # keep unknown until real trials category = "security" # must be an allowed active category subcategories = ["tool_specific"] # or [] if none fit number_of_milestones = 0 # 0 for normal tasks — still required codebase_size = "minimal" # minimal | small | large languages = ["bash"] # agent/oracle languages tags = ["openssl", "certs", "cli"] # 3–6 keywords expert_time_estimate_min = 45 junior_time_estimate_min = 120 [verifier] timeout_sec = 450.0 # how long tests may run [agent] timeout_sec = 900.0 # how long the agent may run [environment] build_timeout_sec = 600.0 # docker build budget cpus = 2 memory_mb = 4096 storage_mb = 10240 allow_internet = false # almost always false
[metadata] — who / what / how hard (all required for Edition 2)
| Field / piece | What it means | How to fill it |
|---|---|---|
| author_name | Who wrote the task | Use anonymous (project standard) unless told otherwise |
| author_email | Contact email | Use anonymous (project standard) |
| difficulty | Declared difficulty before/after trials | Before evals: unknown. Allowed: unknown | easy | medium | hard |
| category | Primary task category | Must be an active category (e.g. security, machine-learning). Banned here: debugging, software-engineering, data-processing |
| subcategories | Optional subtypes the task also fits | Array. Can be empty []. Examples: long_context, tool_specific, api_integration, db_interaction, ui_building. Multiple OK when true |
| number_of_milestones | How many milestone steps the task has | Use 0 if it is a normal (non-milestone) task. Still required even when there are no milestones |
| codebase_size | How big the agent-visible environment is | minimal ≈ 0–20 files · small ≈ 20+ · large ≈ 200+. Count files in environment the agent operates on — not files the agent creates. Be honest; don’t pad |
| languages | Languages the agent writes / oracle uses | Array, e.g. ["bash"], ["r"], ["minizinc"]. Describe agent-facing solution language, not verifier Python helpers |
| tags | Free-form keywords for tools/topics | Required count: about 3–6. One tag alone is not enough. For tool/api/db subtypes, include the specific tool name |
| expert_time_estimate_min | Minutes an expert would need | Integer minutes — your estimate for a careful expert |
| junior_time_estimate_min | Minutes a junior would need | Integer minutes — usually higher than expert |
[verifier] / [agent] / [environment] — run limits & resources
| Field / piece | What it means | How to fill it |
|---|---|---|
| [verifier].timeout_sec | Max seconds test.sh may run | Required. Platform cap commonly 1800. Often ≥ agent timeout |
| [agent].timeout_sec | Max seconds the agent may work | Required. Platform cap commonly 1800 |
| [environment].build_timeout_sec | Max seconds to build the Docker image | Required |
| cpus | CPU cores for the sandbox | Required integer, e.g. 2 |
| memory_mb | RAM in megabytes | Required, e.g. 4096 |
| storage_mb | Disk budget in megabytes | Required, e.g. 10240 |
| allow_internet | Can the agent reach the network at runtime? | Required. Almost always false — bake deps into the image |
| gpus / gpu_types | GPU allocation | Optional — only if the task truly needs a GPU |
| docker_flags | Extra docker run flags | Optional — never use privileged / dangerous caps |
| workdir | Shared working directory | Milestone tasks only — omit on standard tasks |
| custom_docker_compose | Task uses compose file | Set true if environment/docker-compose.yaml exists |
| is_multi_container | More than one service | Set true with custom_docker_compose when compose defines multiple services — both flags together |
instruction.md — what each rule means
Not a TOML file — it’s free prose. Still has hard rules reviewers and the assessment care about.
Instruction principles (treat like fields)
| Field / piece | What it means | How to fill it |
|---|---|---|
| Audience | Who reads it? | Only the AI agent. It does not see task.toml, tests/, or solution/ |
| Style | How should it sound? | Like a real engineer prompting a coding agent — natural, maybe imperfect. Not a heavy markdown brochure |
| Length | How long? | Concise: roughly one sentence up to a few short paragraphs. Dozens of requirement bullets usually means padded, not harder |
| Well-specified | What must be clear? | Every behavior tests grade must be stated (or clearly implied via a referenced spec) |
| Absolute paths | How to mention files? | Always full paths (e.g. /app/config/settings.json). Relative “in the configs folder” fails |
| No hints / recipes | What is forbidden? | Step-by-step “first do X then Y” solution walks, answer values, grader secrets |
| Interesting | Why does it exist? | Must be useful/interesting to some developers — not obscure for obscurity’s sake |
| Specs in environment/ | Can I add docs? | Yes — as realistic contracts/schemas. Not as a second instruction.md to dodge length limits, and not as a solution guide |
environment/ — Dockerfile pieces explained
Dockerfile directives & policy
| Field / piece | What it means | How to fill it |
|---|---|---|
| FROM …@sha256:… | Base image for a build stage | Every FROM must be digest-pinned. Final stage should be a canonical approved image |
| WORKDIR | Default directory in the container | Usually /app — where the agent works |
| RUN apt-get … tmux asciinema | Install runtime tools | tmux + asciinema are required by the agent runtime — missing them breaks agent runs |
| RUN pip/npm install …==version | App dependencies | Pin versions. Apt package version pins are the usual exception |
| COPY app/ /app/ | Ship starter files into the image | Copy only agent-visible content from environment/ build context |
| COPY . . | Copy everything | FORBIDDEN — leaks tests/ and solution/ |
| CMD / USER / --platform | Startup / user / arch lock | FORBIDDEN — Harbor controls entrypoint; verifier needs root; don’t lock arch |
| .dockerignore | Files excluded from build context | Must list solution/, tests/, plus caches (.git, node_modules, __pycache__, …) |
| docker-compose.yaml | Multi-service stack | Optional. If present with 2+ services → set custom_docker_compose=true AND is_multi_container=true |
| Size limits | How big can environment be? | About ≤100 MiB total, ≤50 MiB per file (policy) |
| Network at build | May Dockerfile download data? | Only package dependencies. Task datasets/content must be stored locally in environment/ |
solution/solve.sh — oracle fields of meaning
What the oracle is responsible for
| Field / piece | What it means | How to fill it |
|---|---|---|
| Role | What is it? | Reference solution Harbor runs with --agent oracle. Copied to /oracle/ at runtime |
| Reward contract | What score? | Must produce reward 1 every single run (deterministic) |
| Method | How should it solve? | Real steps that derive the answer — not echo/cat a hardcoded final file |
| Dependencies | Can it install packages? | No internet / no pip install mid-run. Everything preinstalled in the image |
| Flakiness | Random / timing? | Forbidden — must pass consistently |
| chmod +x | Executable bit | Script should be executable (or invoked via bash explicitly) |
tests/ — verifier pieces explained
test.sh + pytest + reward
| Field / piece | What it means | How to fill it |
|---|---|---|
| tests/test.sh | Verifier entrypoint | Always required — even if you also have test_outputs.py. Platform expects this name |
| test_outputs.py (test_*.py) | Actual assertions | Python pytest. Recompute correctness; don’t only check “file exists” |
| /logs/verifier/reward.txt | The grade file | Must write binary 0 or 1. Always write it — even when tests crash |
| Partial scores | 0.7 for 7/10? | Not allowed — binary only |
| Identical logic | Oracle vs agent | Same checks — no “if oracle then relax” branches |
| Instruction alignment | What may tests check? | Only requirements stated (or clearly implied) in instruction.md / referenced specs |
| Hidden instances | Extra datasets | Optional hardening: re-run agent program on secret data (disclose that it happens, not the data) |
| set -e / heredocs | Shell pitfalls | Forbidden in test.sh — can skip reward write or break runners |
| Network in test.sh | Download jq / pip? | Forbidden — preinstall in Dockerfile or vendored wheels |
| Reward-writing tail | Formatting | Don’t restyle — platform pattern-matches the reward write |
Golden contracts
- Oracle run → reward 1
- Nop run → reward 0
- Reward file always written
harbor run -p <task> -a oracle # expect 1 harbor run -p <task> -a nop # expect 0
Rubric — not a Harbor file, but assessment-critical
Created in the Snorkel UI (generate, then edit). Not a required file inside the task zip.
Rubric line anatomy
| Field / piece | What it means | How to fill it |
|---|---|---|
| Where | Create/edit location | Submission UI rubric generator — keep Generate rubrics enabled |
| Line shape | How each criterion looks | Starts with “Agent …”, then a signed score. Example: Agent writes tests for edge cases, +2 |
| Positive scores | Good behaviors | Use explicit + (+1,+2,+3,+5). Sum of all positives must be between 10 and 40 |
| Negative scores | Bad behaviors | At least 3 negative criteria (−1/−2/−3/−5) for harmful or incorrect behavior |
| Phrasing | Positive vs negative wording | Score the event that happened: prefer “Agent accesses /secret, −1” over “Agent does not access…, +1” |
| No /tests references | Why? | Tests run after the attempt — rubrics grade the agent trace, not pytest results |
| No instruction.md / task.toml by filename | Why? | Agent doesn’t “know” those filenames the way authors do — describe content/behavior instead |
| One big rubrics.txt for milestones | OK? | No — handle rubrics through the UI, not a single offline document for all milestones |