Public benchmarks¶
lcode bench has tasks of its own that say which model works best on your machine. To
measure lcode against published results and other agents, it also runs on two public benchmarks
that anyone can rerun:
- Terminal-Bench 2.0: tasks in a terminal, from fixing a git repository to building software, recovering data or setting up servers.
- SWE-bench Verified: real bug fixes in Python projects such as Django, SymPy and matplotlib, checked by the project's own tests.
Both run on Harbor. Harbor starts each task in a
fresh Docker container. lcode is installed in that container and works on the task as one request
(lcode -p "<the task>" --yolo), with the model on your machine. Afterwards the task's tests decide
whether it passed.
What you need¶
- Docker, running.
- uv: lcode runs Harbor with
uvx, so there's nothing else to install. Harbor runs from uv's cache, so don't clear it during a run (uv cache clean): the attempts still to come can't start. - Disk space for the tasks' images: about 20 GB for the dev slice. Each SWE-bench task has an
image of its own, 1–3 GB. Keep about 25 GB free beyond that while it runs: an attempt at
torch-tensor-parallelismtakes about 21 GB until it ends, because the model and then the task's tests each install torch with its GPU libraries. On a smaller disk, run the other tasks with--tasks(lcode bench --harbor dev --listshows them). - A model server the containers can reach. lcode in a container finds your machine by itself:
as
host.docker.internalwith Docker Desktop (macOS, Windows), or as the container's gateway on Linux. Ollama has to listen on more than127.0.0.1for that:- Ollama in Docker with its port published (
-p 11434:11434) already does. - Ollama installed on Linux listens only on
127.0.0.1. Runsudo systemctl edit ollama, addEnvironment="OLLAMA_HOST=0.0.0.0"under[Service], and restart it. That also opens it to your network, so keep port 11434 closed in your firewall.
- Ollama in Docker with its port published (
Running it¶
lcode bench --harbor dev # the dev slice with your model, 3 attempts per task
lcode bench --harbor dev qwen3.6-35b --context 64k
lcode bench --harbor dev --tasks cobol-modernization,django__django-11555 --attempts 1
lcode bench --harbor dev --list # the slice's tasks
Tasks run one at a time, since they share your GPU. The context is your configured one unless you
give --context. A container can't see your GPU, so lcode there doesn't choose a context itself.
lcode in the container is the same version as yours: from PyPI, or a wheel built from your checkout when you run lcode from one, so a change can be measured before it's released.
The slices¶
| Slice | Terminal-Bench 2.0 | SWE-bench Verified | For |
|---|---|---|---|
dev |
15 | 10 | everyday checks of a change |
held-out |
14 | 11 | confirming a change helps beyond the tasks it was tuned on |
Both were drawn once, at random and by difficulty, from the tasks a local model can try in a night:
- Terminal-Bench without its model-training tasks, and with at most 30 minutes per task: 2 easy, 10 medium and 3 hard tasks in the dev slice.
- SWE-bench Verified's two common difficulties: five fixes estimated under 15 minutes and five of 15 minutes to an hour.
Don't tune lcode on the held-out set: once a change is chosen because it does well there, the set no longer tells you whether changes generalize.
What's reported¶
Two tasks, one attempt each, with qwen3.6-35b at 32K on a 12 GB laptop GPU:
$ lcode bench --harbor dev --tasks prove-plus-comm,django__django-11555 --attempts 1
…
Task Benchmark Passed Time Tokens in Tokens out
prove-plus-comm terminal-bench 1/1 0.6 min 27,827 1,494
django__django-11555 swebench-verified 0/1 19.9 min 2,370,217 38,508
pass@1 50% · pass^1 50%
With three attempts, the last line also gives each run's pass rate and the spread between them.
- pass@1: the share of attempts that passed, averaged over the tasks. It's the number to compare with leaderboards.
- pass^3: the share of tasks whose three attempts all passed. That shows how dependable lcode is, not just how often it gets lucky.
- runs and spread: the pass rate of each task's first attempt, its second and its third. Single
runs of the same setup differ by several points;
--compare(below) says whether a change between two results is more than that. - Time and tokens per attempt, from lcode's own record of the request (its
trace). An attempt stopped at the task's time limit counts the tokens recorded
until then, and a reply lcode stopped counts with an estimate.
–means no attempt of the task has a record. - The model's speed: how many tokens per second it generated, in the median attempt, from the same traces. A slower machine (short of memory, or busy with other programs) runs into the tasks' time limits more often.
Attempts whose container didn't start (a failed image download, say) count neither as passed nor as failed. Attempts whose tests ran but didn't finish (a time limit, say) or gave no result count as failed, as in Harbor's own mean, so a change that makes the tests hang can't hide a drop. The report shows how many of each there were.
Everything is saved in ~/.local/state/lcode/bench/<slice>-<time>/:
lcode-bench.json: the results, with the lcode and Harbor versions, the model, the context, the machine, when the run started and how long it took, and the command that reproduces them.--json FILEsaves a copy.- Harbor's own results: each attempt's log, the task's test output, and lcode's trace and session.
uvx --from harbor harbor view <folder>browses them.
Comparing two runs¶
lcode bench --compare compares two saved results, for example the dev slice before and after a
change, given as their folders or their lcode-bench.json:
lcode bench --compare ~/.local/state/lcode/bench/dev-20261101-161623 ~/.local/state/lcode/bench/dev-20261105-201500
It says which lcode, model and context each one ran, warns when they don't match, lists the tasks
whose results changed (3/3 → 1/3), and compares pass@1 over the tasks both ran: within the noise,
better, or a drop beyond the noise.
While Harbor's results are still on the machine, --compare reads both runs from them again, so a
run saved by an older lcode counts its attempts the same way as a new one. It also gives each run's
model speed, and warns when one ran more than 10% slower: more of its attempts run into their time
limits, so part of a difference can come from the machine rather than from lcode.
The noise is how far pass@1 moves between runs of the same setup, estimated from how much each
task's attempts disagree: a task that always passes or always fails adds nothing, one that passes
half the time the most. One attempt can't disagree with itself, so that task's pass rate comes from
both runs. A change counts when it's more than twice the standard error of the difference. On the dev
slice, with three attempts per task, that's about 9 points. --compare warns when the two runs didn't
have the same number of attempts (a container that didn't start, say), and exits with 1 when the second
result dropped beyond the noise, so a script can stop on it.
The release rule: a release that changes lcode's prompts, tools or context handling ships only if the dev slice doesn't drop beyond the noise against the last baseline, and the comparison goes on its pull request. See the release steps.
Baselines¶
lcode 0.63.0 on the dev slice, on a laptop with a 12 GB GPU (RTX 4080 Laptop) and 31 GB of RAM, with nothing else running:
| Model | Context | Attempts | pass@1 | Terminal-Bench | SWE-bench | pass^3 | Speed | Time |
|---|---|---|---|---|---|---|---|---|
| qwen3.6-35b | 32K | 3 | 55% | 51% | 60% | 36% | 73 tokens/s | 11.4 h |
| qwen3.6-35b | 64K | 3 | 61% | 51% | 77% | 40% | 67 tokens/s | 11.8 h |
| qwen3.6-35b | 256K | – | stopped after 2 h | 26–46 tokens/s | ||||
| qwen3.8-27b | 32K | 1 | 36% of 22 tasks | 20% | 71% of 7 | – | 9 tokens/s | stopped after 9 h |
To reproduce them:
lcode bench --harbor dev qwen3.6-35b --context 32k
lcode bench --harbor dev qwen3.6-35b --context 64k
lcode bench --harbor dev qwen3.6-35b --context 256k
lcode bench --harbor dev qwen3.8-27b --context 32k --attempts 1
- 32K or 64K: 64K passed 7 points more, within the noise (±11), so this slice doesn't show that 64K helps. It did waste less: 5 replies that went nowhere instead of 20. The 32K run is the baseline the release rule compares against.
- 256K doesn't fit this machine. Ollama kept 4 of the model's 9 GB on the GPU and took 23 GB of
RAM, so the system started swapping. Generation fell from 40–46 to 26 tokens/s, and the run was
stopped after 2 hours. By then 2 of 8 Terminal-Bench attempts had passed and 6 had run out of time,
crack-7z-hashamong them, which passed 3 times out of 3 at 32K. With 12 GB of GPU memory, use 64K at most. - qwen3.8-27b is too slow on a 12 GB GPU. The dense 27B model is larger than the GPU (8 of its 20 GB fit), so it generates 9 tokens/s, and 12 of the 15 Terminal-Bench tasks ran out of time. The run was stopped after 9 hours, when generation fell to 4 tokens/s, and 3 SWE-bench tasks didn't run. On the 22 tasks it ran, qwen3.6-35b at 32K passed 56%.
Runs on other machines differ, mostly through the model's speed: a faster GPU runs into the tasks' time limits less often.
Other agents on the same model¶
--agent runs Claude Code, OpenCode or Codex on the same tasks, with the same model, context
window and time limits, so you can compare them with lcode:
lcode bench --harbor dev qwen3.6-35b --context 64k # lcode
lcode bench --harbor dev qwen3.6-35b --context 64k --agent claude-code
lcode bench --harbor dev qwen3.6-35b --context 64k --agent opencode
lcode bench --harbor dev qwen3.6-35b --context 64k --agent codex
Harbor installs the agent in each task's container, with approvals off. lcode sets the agent up the
way ollama launch does, with the Ollama on your machine:
- Claude Code uses Ollama's Anthropic-compatible API, with the model for every tier (subagents
too) and
ollama launch claude's settings: no telemetry, error reports or feedback prompts. Likeollama launch claude, it doesn't tell Claude Code the context window. - OpenCode uses an Ollama provider on Ollama's OpenAI-compatible API. The model is marked as a reasoning model, with the limits of its context window.
- Codex uses Ollama's Responses API and
ollama launch codex's model catalog. That catalog gives local models no base instructions, so Codex runs without its own system prompt, as it does withollama launch codex.
These agents' APIs can't set the context window per request, and Ollama's default is small (4K on
GPUs under 24 GB), so lcode makes a copy of the model with the context set:
lcode-qwen3.6-35b-64k for 64K. It uses the model's files, so nothing is downloaded, and
ollama rm lcode-qwen3.6-35b-64k removes it. Ollama recommends at least 64K for these agents.
The time per task leaves out installing the agent, which Harbor does in every task's container: up to
three minutes for Claude Code, so its runs take longer overall. The results go to
~/.local/state/lcode/bench/<slice>-<agent>-<time>/, and lcode-bench.json adds the agent and its
version.
With Harbor directly¶
lcode.harbor:Lcode is an ordinary Harbor agent, so every Harbor dataset and option works:
uvx --python 3.13 --from harbor==0.25.0 --with lcode-cli harbor run \
-d terminal-bench@2.0 -a lcode.harbor:Lcode -m qwen3.6-35b --ak context=32768
Its options, each given with --ak name=value:
| Option | What it does |
|---|---|
context |
the context window, e.g. 32768 |
host |
Ollama's address as the container sees it, e.g. http://192.168.1.20:11434 |
version |
the lcode version to install (default: the one Harbor runs with) |
wheel |
an lcode wheel on your machine to install instead |
Other model servers work as in lcode itself: --ae LCODE_BACKEND=openai --ae LCODE_BASE_URL=….
When Docker can't make networks¶
Harbor gives each task a network of its own. When Docker says all predefined address pools have
been fully subnetted, every address range it could use overlaps a route on your machine. A VPN that
sends all traffic through itself does that. Add --docker-bridge, and the tasks run on Docker's
default network instead. With Harbor directly, pass --extra-docker-compose with a file holding: