Snorkel Terminal-Bench CLI

macOS + LinuxPython 3.12+

Build, test, submit, and review with STB

The complete command-line path from first install to accepted task—including revisions, reviews, adjudications, status meanings, and common failures.

Operating system

macOS or Linux

Windows is not supported.

Runtime

Docker running

The CLI must reach the Docker socket.

Toolchain

uv + Python 3.12+

uv installs and isolates STB globally.

01 / STB guide

Install and authenticate

Run these once on a supported machine. Keep Docker Desktop open while using STB.

Verify Docker

Install Docker Desktop or Docker Engine first. STB does not start Docker for you.

docker version
docker run --rm hello-world

Install uv

uv is the package manager used to install the STB command globally.

curl -LsSf https://astral.sh/uv/install.sh | sh
uv --version

Install STB

The package is served from Snorkel’s wheel index and requires Python 3.12 or newer.

uv tool install snorkelai-stb \
  --find-links https://snorkel-python-wheels.s3.us-west-2.amazonaws.com/stb/index.html \
  --python ">=3.12"

Verify the CLI

Restart the terminal if the command is not immediately available.

stb --version
stb --help

Sign in

Generate an API key on the Experts platform, then paste it into the terminal prompt.

stb login

Check AI credentials

View the current credentials or refresh them when expired or exhausted.

stb keys show
stb keys refresh

02 / STB guide

Choose your workflow

The same CLI covers task authors, reviewers, and adjudicators.

Initialize a task, test it with the oracle and real agents, then submit it.

01
Find your project
Confirm that the Submitter role is active. You can use the project name or UUID in later commands.
```
stb projects list
```
02
Initialize the task
Choose the template assigned to your project. The template flag is optional when only one template exists.
```
stb init my-task-name -p PROJECT_ID -t default
# Milestone task:
stb init my-task-name -p PROJECT_ID -t milestone
```

Develop and test

Edit the generated task files, run the oracle, then run each target model two or three times to estimate difficulty.

stb harbor tasks start-env -p ./my-task-name -i
stb harbor run -a oracle -p ./my-task-name
stb harbor run -m @openai/gpt-5.5 -p ./my-task-name
stb harbor run -m @anthropic/claude-opus-4-8 -p ./my-task-name

04
Configure the rubric in the UI
The CLI does not create rubrics. Open the task on the Experts platform and finish the rubric before submission.
05
Submit
Record actual working time in minutes. STB zips the folder, uploads it, runs checks, and creates the submission.
```
stb submissions create ./my-task-name -p PROJECT_ID --time 120
```

Track and inspect

List project submissions, open one in the browser, download its files, or retrieve the latest feedback.

stb submissions list -p PROJECT_ID --show-folder-names
stb submissions view SUBMISSION_ID
stb submissions feedback SUBMISSION_ID

03 / STB guide

Read submission status

Only NEEDS_REVISION permits an update. Other states are informational or final.

Status	Meaning	Can update?
EVALUATION_PENDING	Automated checks are running	No
NEEDS_REVISION	Reviewer requested changes	Yes
REVIEW_PENDING	Waiting for a human reviewer	No
ACCEPTED	Task accepted	No
OFFERED	An offer was made	No
REJECTED	Task rejected	No
SKIPPED	Task skipped	No

04 / STB guide

Command reference

Use placeholders such as PROJECT_ID and SUBMISSION_ID exactly where shown.

Account

Sign in: stb login
Show AI credentials: stb keys show
Refresh AI credentials: stb keys refresh
Launch configured Claude Code: stb claude

Projects & tasks

List assignments: stb projects list
Create a task: stb init TASK_NAME -p PROJECT_ID -t TEMPLATE_NAME
Open an interactive environment: stb harbor tasks start-env -p ./TASK_NAME -i
Run the oracle: stb harbor run -a oracle -p ./TASK_NAME

Submissions

Create: stb submissions create ./TASK_NAME -p PROJECT_ID --time MINUTES
List: stb submissions list -p PROJECT_ID
Open: stb submissions view SUBMISSION_ID
Download: stb submissions download SUBMISSION_ID
Feedback: stb submissions feedback SUBMISSION_ID
Update: stb submissions update ./TASK_NAME -s SUBMISSION_ID --time MINUTES

Upgrade STB

uv tool install snorkelai-stb \
  --find-links https://snorkel-python-wheels.s3.us-west-2.amazonaws.com/stb/index.html \
  --python ">=3.12" --reinstall --no-cache

Configuration and credentials are preserved.

Uninstall STB

uv tool uninstall snorkelai-stb
# Optional: remove local config
rm -rf ~/.config/stb/

Removing the config signs this machine out and deletes local settings.

05 / STB guide

Troubleshooting & FAQ

Search by an error, command, status, or symptom.

What should I do if I see E006 or STB cannot connect to Docker?

Treat it as a Docker connectivity problem first. Start Docker Desktop, verify the Docker CLI can reach the engine, and retry STB. On macOS, enable “Allow the default Docker socket to be used” in Docker Desktop Advanced settings. E006 is not defined in public STB documentation, so include the full command and error output when escalating.

docker version
docker ps
docker run --rm hello-world

Why is the stb command not found after installation?

The uv tool bin directory is probably not in the current shell PATH. Update the shell configuration, then restart the terminal.

uv tool update-shell
# Restart the terminal, then:
stb --version

Why does STB say this folder has already been submitted?

A submitted folder is linked to an existing submission. Use submissions update instead of creating a second submission.

stb submissions update ./my-task-name --time 60

Why can I not update this submission?

Updates are allowed only in NEEDS_REVISION. Run the list command to confirm the current state. Do not create a replacement submission to bypass the state.

stb submissions list -p PROJECT_ID

What does “Maximum refresh limit reached” mean?

The account has reached its AI credential refresh cap. Contact an admin to reset or raise the limit, then run the refresh command again.

stb keys refresh

Can I ignore “Failed to retrieve model info”?

Yes, when it is the known model-information warning and the run continues. It does not mean the model execution itself failed. Investigate only if the run stops or reports a separate authentication error.

Why does the milestone scaffold have root-level solveN.sh files?

That is the older milestone layout. Upgrade STB before continuing. The current milestone template uses self-contained subdirectories under steps/.

uv tool install snorkelai-stb --find-links https://snorkel-python-wheels.s3.us-west-2.amazonaws.com/stb/index.html --python ">=3.12" --reinstall --no-cache

Does STB configure the rubric during submission?

No. Complete the rubric in the Experts platform UI before running submissions create. STB packages and uploads the task but does not include a rubric step.

Is Windows supported?

No. STB currently does not support Windows. Use macOS or Linux rather than relying on a Windows-native setup.

How do I collect useful information before asking for help?

Record the STB version, exact command, full error output, Docker status, operating system, project ID, and submission or review ID. Never paste API keys or AI credentials into Slack or an issue.

stb --version
docker version
stb projects list

Keep learning

Understand the files before you submit.

STB handles the workflow. The Task and Craft guides explain what a strong Terminal-Bench task must contain and how reviewers judge it.

Read task anatomy Review craft rules