Skip to content

Public benchmarks

lcode bench has tasks of its own that say which model works best on your machine. To measure lcode against published results and other agents, it also runs on two public benchmarks that anyone can rerun:

  • Terminal-Bench 2.0: tasks in a terminal, from fixing a git repository to building software, recovering data or setting up servers.
  • SWE-bench Verified: real bug fixes in Python projects such as Django, SymPy and matplotlib, checked by the project's own tests.

Both run on Harbor. Harbor starts each task in a fresh Docker container. lcode is installed in that container and works on the task as one request (lcode -p "<the task>" --yolo), with the model on your machine. Afterwards the task's tests decide whether it passed.

What you need

  • Docker, running.
  • uv: lcode runs Harbor with uvx, so there's nothing else to install. Harbor runs from uv's cache, so don't clear it during a run (uv cache clean): the attempts still to come can't start.
  • Disk space for the tasks' images: about 20 GB for the dev slice. Each SWE-bench task has an image of its own, 1–3 GB. Keep about 25 GB free beyond that while it runs: an attempt at torch-tensor-parallelism takes about 21 GB until it ends, because the model and then the task's tests each install torch with its GPU libraries. On a smaller disk, run the other tasks with --tasks (lcode bench --harbor dev --list shows them).
  • A model server the containers can reach. lcode in a container finds your machine by itself: as host.docker.internal with Docker Desktop (macOS, Windows), or as the container's gateway on Linux. Ollama has to listen on more than 127.0.0.1 for that:
    • Ollama in Docker with its port published (-p 11434:11434) already does.
    • Ollama installed on Linux listens only on 127.0.0.1. Run sudo systemctl edit ollama, add Environment="OLLAMA_HOST=0.0.0.0" under [Service], and restart it. That also opens it to your network, so keep port 11434 closed in your firewall.

Running it

lcode bench --harbor dev                       # the dev slice with your model, 3 attempts per task
lcode bench --harbor dev qwen3.6-35b --context 64k
lcode bench --harbor dev --tasks cobol-modernization,django__django-11555 --attempts 1
lcode bench --harbor dev --list                # the slice's tasks

Tasks run one at a time, since they share your GPU. The context is your configured one unless you give --context. A container can't see your GPU, so lcode there doesn't choose a context itself.

lcode in the container is the same version as yours: from PyPI, or a wheel built from your checkout when you run lcode from one, so a change can be measured before it's released.

The slices

Slice Terminal-Bench 2.0 SWE-bench Verified For
dev 15 10 everyday checks of a change
held-out 14 11 confirming a change helps beyond the tasks it was tuned on

Both were drawn once, at random and by difficulty, from the tasks a local model can try in a night:

  • Terminal-Bench without its model-training tasks, and with at most 30 minutes per task: 2 easy, 10 medium and 3 hard tasks in the dev slice.
  • SWE-bench Verified's two common difficulties: five fixes estimated under 15 minutes and five of 15 minutes to an hour.

Don't tune lcode on the held-out set: once a change is chosen because it does well there, the set no longer tells you whether changes generalize.

What's reported

Two tasks, one attempt each, with qwen3.6-35b at 32K on a 12 GB laptop GPU:

$ lcode bench --harbor dev --tasks prove-plus-comm,django__django-11555 --attempts 1
…
Task                  Benchmark          Passed      Time  Tokens in  Tokens out
prove-plus-comm       terminal-bench        1/1   0.6 min     27,827       1,494
django__django-11555  swebench-verified     0/1  19.9 min  2,370,217      38,508
pass@1 50% · pass^1 50%

With three attempts, the last line also gives each run's pass rate and the spread between them.

  • pass@1: the share of attempts that passed, averaged over the tasks. It's the number to compare with leaderboards.
  • pass^3: the share of tasks whose three attempts all passed. That shows how dependable lcode is, not just how often it gets lucky.
  • runs and spread: the pass rate of each task's first attempt, its second and its third. Single runs of the same setup differ by several points; --compare (below) says whether a change between two results is more than that.
  • Time and tokens per attempt, from lcode's own record of the request (its trace). An attempt stopped at the task's time limit counts the tokens recorded until then, and a reply lcode stopped counts with an estimate. – means no attempt of the task has a record.
  • The model's speed: how many tokens per second it generated, in the median attempt, from the same traces. A slower machine (short of memory, or busy with other programs) runs into the tasks' time limits more often.

Attempts whose container didn't start (a failed image download, say) count neither as passed nor as failed. Attempts whose tests ran but didn't finish (a time limit, say) or gave no result count as failed, as in Harbor's own mean, so a change that makes the tests hang can't hide a drop. The report shows how many of each there were.

Everything is saved in ~/.local/state/lcode/bench/<slice>-<time>/:

  • lcode-bench.json: the results, with the lcode and Harbor versions, the model, the context, the machine, when the run started and how long it took, and the command that reproduces them. --json FILE saves a copy.
  • Harbor's own results: each attempt's log, the task's test output, and lcode's trace and session. uvx --from harbor harbor view <folder> browses them.

Comparing two runs

lcode bench --compare compares two saved results, for example the dev slice before and after a change, given as their folders or their lcode-bench.json:

lcode bench --compare ~/.local/state/lcode/bench/dev-20261101-161623 ~/.local/state/lcode/bench/dev-20261105-201500

It says which lcode, model and context each one ran, warns when they don't match, lists the tasks whose results changed (3/3 → 1/3), and compares pass@1 over the tasks both ran: within the noise, better, or a drop beyond the noise.

While Harbor's results are still on the machine, --compare reads both runs from them again, so a run saved by an older lcode counts its attempts the same way as a new one. It also gives each run's model speed, and warns when one ran more than 10% slower: more of its attempts run into their time limits, so part of a difference can come from the machine rather than from lcode.

The noise is how far pass@1 moves between runs of the same setup, estimated from how much each task's attempts disagree: a task that always passes or always fails adds nothing, one that passes half the time the most. One attempt can't disagree with itself, so that task's pass rate comes from both runs. A change counts when it's more than twice the standard error of the difference. On the dev slice, with three attempts per task, that's about 9 points. --compare warns when the two runs didn't have the same number of attempts (a container that didn't start, say), and exits with 1 when the second result dropped beyond the noise, so a script can stop on it.

The release rule: a release that changes lcode's prompts, tools or context handling ships only if the dev slice doesn't drop beyond the noise against the last baseline, and the comparison goes on its pull request. See the release steps.

Baselines

lcode 0.63.0 on the dev slice, on a laptop with a 12 GB GPU (RTX 4080 Laptop) and 31 GB of RAM, with nothing else running:

Model Context Attempts pass@1 Terminal-Bench SWE-bench pass^3 Speed Time
qwen3.6-35b 32K 3 55% 51% 60% 36% 73 tokens/s 11.4 h
qwen3.6-35b 64K 3 61% 51% 77% 40% 67 tokens/s 11.8 h
qwen3.6-35b 256K – stopped after 2 h 26–46 tokens/s
qwen3.8-27b 32K 1 36% of 22 tasks 20% 71% of 7 – 9 tokens/s stopped after 9 h

To reproduce them:

lcode bench --harbor dev qwen3.6-35b --context 32k
lcode bench --harbor dev qwen3.6-35b --context 64k
lcode bench --harbor dev qwen3.6-35b --context 256k
lcode bench --harbor dev qwen3.8-27b --context 32k --attempts 1
  • 32K or 64K: 64K passed 7 points more, within the noise (±11), so this slice doesn't show that 64K helps. It did waste less: 5 replies that went nowhere instead of 20. The 32K run is the baseline the release rule compares against.
  • 256K doesn't fit this machine. Ollama kept 4 of the model's 9 GB on the GPU and took 23 GB of RAM, so the system started swapping. Generation fell from 40–46 to 26 tokens/s, and the run was stopped after 2 hours. By then 2 of 8 Terminal-Bench attempts had passed and 6 had run out of time, crack-7z-hash among them, which passed 3 times out of 3 at 32K. With 12 GB of GPU memory, use 64K at most.
  • qwen3.8-27b is too slow on a 12 GB GPU. The dense 27B model is larger than the GPU (8 of its 20 GB fit), so it generates 9 tokens/s, and 12 of the 15 Terminal-Bench tasks ran out of time. The run was stopped after 9 hours, when generation fell to 4 tokens/s, and 3 SWE-bench tasks didn't run. On the 22 tasks it ran, qwen3.6-35b at 32K passed 56%.

Runs on other machines differ, mostly through the model's speed: a faster GPU runs into the tasks' time limits less often.

Other agents on the same model

--agent runs Claude Code, OpenCode or Codex on the same tasks, with the same model, context window and time limits, so you can compare them with lcode:

lcode bench --harbor dev qwen3.6-35b --context 64k                      # lcode
lcode bench --harbor dev qwen3.6-35b --context 64k --agent claude-code
lcode bench --harbor dev qwen3.6-35b --context 64k --agent opencode
lcode bench --harbor dev qwen3.6-35b --context 64k --agent codex

Harbor installs the agent in each task's container, with approvals off. lcode sets the agent up the way ollama launch does, with the Ollama on your machine:

  • Claude Code uses Ollama's Anthropic-compatible API, with the model for every tier (subagents too) and ollama launch claude's settings: no telemetry, error reports or feedback prompts. Like ollama launch claude, it doesn't tell Claude Code the context window.
  • OpenCode uses an Ollama provider on Ollama's OpenAI-compatible API. The model is marked as a reasoning model, with the limits of its context window.
  • Codex uses Ollama's Responses API and ollama launch codex's model catalog. That catalog gives local models no base instructions, so Codex runs without its own system prompt, as it does with ollama launch codex.

These agents' APIs can't set the context window per request, and Ollama's default is small (4K on GPUs under 24 GB), so lcode makes a copy of the model with the context set: lcode-qwen3.6-35b-64k for 64K. It uses the model's files, so nothing is downloaded, and ollama rm lcode-qwen3.6-35b-64k removes it. Ollama recommends at least 64K for these agents.

The time per task leaves out installing the agent, which Harbor does in every task's container: up to three minutes for Claude Code, so its runs take longer overall. The results go to ~/.local/state/lcode/bench/<slice>-<agent>-<time>/, and lcode-bench.json adds the agent and its version.

With Harbor directly

lcode.harbor:Lcode is an ordinary Harbor agent, so every Harbor dataset and option works:

uvx --python 3.13 --from harbor==0.25.0 --with lcode-cli harbor run \
  -d terminal-bench@2.0 -a lcode.harbor:Lcode -m qwen3.6-35b --ak context=32768

Its options, each given with --ak name=value:

Option What it does
context the context window, e.g. 32768
host Ollama's address as the container sees it, e.g. http://192.168.1.20:11434
version the lcode version to install (default: the one Harbor runs with)
wheel an lcode wheel on your machine to install instead

Other model servers work as in lcode itself: --ae LCODE_BACKEND=openai --ae LCODE_BASE_URL=….

When Docker can't make networks

Harbor gives each task a network of its own. When Docker says all predefined address pools have been fully subnetted, every address range it could use overlaps a route on your machine. A VPN that sends all traffic through itself does that. Add --docker-bridge, and the tasks run on Docker's default network instead. With Harbor directly, pass --extra-docker-compose with a file holding:

services:
  main:
    network_mode: bridge