A · Brand-new task
- Pick track with your lead
- discover → pick the idea with the best trap
- research (if the track has it)
- convert → validate (oracle=1, nop=0)
- /local-checks + /human-review
- /snorkel-zip → new submission on the website
Day-to-day
Your daily loop (with a picture)
Two workflows: create something new, or revise a Snorkel submission. Most days you revise.
Pipeline
How a task moves from idea → accepted
Read in order. Loops mean go back and fix. Step 08 is done.
01
Pick a track
Your lead assigns the kind of reasoning (optmodel, datalake, vulns…).
02
Discover ideas
Type /<track>-discover — get 3 ideas. Pick the one with the best trap.
03
Research
/<track>-research turns one idea into a Go / No-Go plan.
04
Convert
/<track>-convert builds the real task folder (Docker, oracle, tests).
05
Validate locally
oracle = 1 · nop = 0 · ruff clean · /local-checks · /human-review
06
Upload to Snorkel
/snorkel-zip → upload ZIP in the web UI → wait for AI eval.
07
Read & revise
Too easy? Harden. Unsolvable? Add fair disclosure. Re-upload same ID.
08
Accepted
HARD (or MEDIUM), Solvable, ≤5/10 passes, reviewer OK → stop.
B · Revise existing
- Open submission page — read difficulty + notes
- Download deployed files; diff vs your branch
- Fix (easy / unsolvable / oracle / reviewer)
- Re-validate everything
- Re-upload ZIP to the same ID · raise AHT
- Only when state is NEEDS_REVISION
| If you see… | It means… | Do this |
|---|---|---|
| Oracle failed | Our solution didn’t score 1 | Fix Docker/oracle first |
| TRIVIAL / EASY | Models solved it too easily | Close leaks; add a hard lever |
| 0/10 unsolvable | Too hard or unfair | Add fair disclosure; keep the trap |
| Needs Revision | Reviewer blocker | Do exactly what they asked |