Assessment prep
Principles reviewers care about — learn judgment, not an answer key. Then try Ask AI.
This file is the machine-readable config for timeouts, resources, tags, and metadata. Treat it like a checklist, not optional fluff. Every task needs agent and verifier timeouts, an environment build timeout, and resource fields (CPU, memory, storage). Even when your task has no intermediate milestones, milestone-related fields the platform expects still need to be present in the expected shape.
instruction.md is the only prose the agent reads. Write the way a real developer would ask a coding agent for help. Skip emoji markers, dense header forests, and “AI brochure” formatting. Short and human beats long and decorative.
Rubrics are created through the Snorkel UI generator, then edited for accuracy. They score what the agent did in its trace — not hidden test internals. Create and edit rubrics in the submission UI. Do not invent a separate rubrics.txt that tries to cover everything offline.
The environment is everything the agent is allowed to see. Keep answers and graders out of it. Package installs should use pinned versions. Apt packages are the usual exception. Do not rely on floating “latest.”
This file is the machine-readable config for timeouts, resources, tags, and metadata. Treat it like a checklist, not optional fluff.
Every task needs agent and verifier timeouts, an environment build timeout, and resource fields (CPU, memory, storage). Even when your task has no intermediate milestones, milestone-related fields the platform expects still need to be present in the expected shape.
One lonely tag is too thin. Aim for a small handful of relevant tags (roughly three to six) that describe languages and challenge areas the agent faces.
If you ship a docker-compose with multiple services, the settings sheet must declare that you are using custom compose and that this is a multi-container task.
Count agent-visible files honestly. A small environment (about 0–20 files) is “minimal.” Do not pad with filler files just to look bigger.
instruction.md is the only prose the agent reads. Write the way a real developer would ask a coding agent for help.
Skip emoji markers, dense header forests, and “AI brochure” formatting. Short and human beats long and decorative.
Most tasks need as little as one sentence up to a few short paragraphs. Long laundry lists of requirements (dozens of bullets) usually mean the task is padded, not harder.
Requirements state outcomes and constraints. “First read X, then parse Y, then filter Z” is a recipe — that is a hint, not a requirement.
If you mention a file or folder, give the full path the agent should use. Vague “update the config in the configs folder” fails review.
Formulaic structure, hedging, and unnaturally balanced pros/cons are red flags. Prompts should mirror how real users talk to coding agents.
Piling on dozens of edge cases to manufacture hardness is not a valid Advanced/Hard task. Genuine reasoning challenge beats checklist length.
A task must matter to some group of developers. Obscure-for-obscurity’s-sake is high severity.
Rubrics are created through the Snorkel UI generator, then edited for accuracy. They score what the agent did in its trace — not hidden test internals.
Create and edit rubrics in the submission UI. Do not invent a separate rubrics.txt that tries to cover everything offline.
Every line starts with “Agent”, describes one behavior, and ends with a signed score. Good behaviors get plus; mistakes get minus.
When you add the positive scores, the total should land in the platform’s accepted window (about 10–40). Too high means revise before submit.
A rubric that only awards positives is incomplete. Include several negative rewards for harmful or incorrect behavior.
Prefer “Agent accesses the secret directory, −1” over “Agent does not access…, +1.” Score the behavior that happened.
Tests run after the attempt. The agent also does not “see” instruction.md as a named file in the way rubrics sometimes imply. Describe content and behavior, not file paths into the grader.
The environment is everything the agent is allowed to see. Keep answers and graders out of it.
Package installs should use pinned versions. Apt packages are the usual exception. Do not rely on floating “latest.”
Only package dependencies may be fetched from the network during build. Datasets and task content belong in the environment locally — not curled from GitHub releases at build time.
Expected answers and grader fixtures live under tests/, never as agent-visible files like expected_output.json in the environment.
Instead of embedding scripts with Dockerfile heredocs, keep files in the build context and COPY them in.
Privileged mode, dangerous capabilities, and mounting docker.sock are prohibited — even if the task “needs” them.
Everything the solution needs must already be in the image. The oracle session does not get internet to pip install mid-run.
The grader must prove real solving happened. The oracle must prove the task is solvable every time.
Write 0 or 1 to reward.txt. Partial credit like 0.7 is not allowed.
Never branch the verifier to go easy on the oracle. Identical checks for both.
Verifying a .json file exists is not enough. Open it and validate the content.
If the prompt never mentions a REST API, do not fail the agent for missing one. Every graded behavior must be stated (or clearly implied) in the instructions or a referenced spec.
If an output file could already sit in the environment, regenerate it by running the agent’s program during verification so a static file cannot sneak a pass.
test.sh should not download packages from the internet. Install verifier dependencies in the Dockerfile (or vendored wheels), including libraries used only by tests.
Even if you have pytest files, test.sh is the required entrypoint that writes the reward.
No flaky random sampling. No hardcoding the final answer with a one-line cat. Derive the answer with real steps, and pass every run.
Difficulty must come from genuine challenge with every required behavior disclosed. Undocumented traps that make agents fail are unfair, not Hard.
Not every coding puzzle is in scope. Aim for realistic agent work with enough steps to matter.
A binary search tree that only lives in memory with no files, services, or tools is out of scope. Data structures can support a larger task — they should not be the whole story.
A single clever shell one-liner is too thin. Tasks should require multiple steps with intermediate state and reasoning.
A huge log file that falls to a simple grep is not a real long-context challenge. The agent should have to reason over the content.
Before packaging, the task parent should contain only the task’s own files — not leftover README noise, old jobs logs, or unrelated data folders.
If you use milestones, instructions must show which requirements belong to which milestone — not one undifferentiated blob.
When a task honestly spans areas (for example long context and API work), you can tag more than one subtype.