Platform literacy
How to read a Snorkel result
Use the result matrix during task creation and submission. Fix real blockers without overreacting to non-blocking statuses.
Task creation and submission process
Interpret the evaluation before acting
| What the result shows | What it means | What to do |
|---|---|---|
| Difficulty MEDIUM or HARD, Status Solvable | Healthy task, in the allowed band and solvable | Submit |
| Difficulty TRIVIAL or EASY | Too easy, below the MEDIUM minimum | Rework to make it harder, keep it solvable, and re-run |
| Red cross, some tests not passed by any agent | Not solvable | Rework the task so the oracle passes every test |
| Oracle not 1, or nop not 0 | The task is broken | Download the agent logs. Give the logs and task to Claude, then ask it to fix why the oracle failed or nop passed |
| AutoEval build update, Build FAILED | Non-blocking status | Ignore it; this status does not stop submission |
| Quality Check pass | Instructions describe what the tests check | Nothing. If it fails, add the missing tested requirement to instruction.md |
| Agent review Warning, Test Quality Robust | Soft suggestion; tests are solid | Ignore |
| Agent review or Test Quality Critical or Vulnerable | Real blocker | Fix it before submitting |
Safe default
Preserve difficulty while fixing correctness
A healthy task is both difficult and solvable. When fixing failures, correct the oracle, environment, tests, or missing disclosures without weakening a legitimate requirement. Re-run oracle and nop after every material change.
Open debugging and revision prompts →Compact quick-reference
| Signal | Do this |
|---|---|
| MEDIUM/HARD + Solvable | Good — submit / send to reviewer when ready |
| TRIVIAL or EASY | Too easy — harden, then re-run |
| Status not Solvable | Fix oracle / tests / instructions |
| AutoEval Build FAILED | Usually ignore (non-blocker) |
| Critical / Vulnerable | Must fix before submit |
| Recommendations under Warning | Resolve them (Warning label alone is OK) |