Benchmarking models¶
lcode bench answers the most common question about local models: which one works best for
coding on my machine? It runs a fixed set of coding tasks against one or more models, three times
each, and reports how many they solved, how fast they ran and how much memory they used.
To compare with other agents on public benchmarks (Terminal-Bench 2.0 and SWE-bench Verified), see Public benchmarks.
lcode bench # the configured model
lcode bench qwen3.6-35b qwen3.5-9b # compare models
lcode bench --tasks fix-bug,rename # run some of the tasks
lcode bench --quick # only the eight quick tasks, once each
lcode bench --attempts 1 # each task once: quicker, but within the noise
lcode bench --json results.json # also save machine-readable results
lcode bench --markdown # also print a table to paste into a test report
lcode bench also runs against other model servers, such as llama-server or LM
Studio: LCODE_BACKEND=llama.cpp lcode bench. The server keeps the model loaded and doesn't report
its memory use, so those columns stay empty.
lcode bench · NVIDIA GeForce RTX 4080 Laptop GPU (12 GB VRAM), 31 GB RAM · Ollama 0.40.13
lcode-qwen3.6-35b · 32K context · reasoning on · 8 tasks · 3 attempts each
Loaded in 18s · ~23 GB, 42% on the GPU and the rest in system RAM · reads prompts at 905 tok/s
Attempt 1 of 3
✓ fix-bug Fix a bug so the tests pass 22s the tests pass
✓ find-code Answer with file:line 8s next_wait at backoff.py:13
✓ write-script Write a script from a spec 12s correct output, also on data it hadn't seen
✓ rename Rename a function across files 22s its tests and extra checks pass
✓ precise-edit One-line edit in a long file 10s changed only service_087's timeout
✓ recover Recover from failing commands 22s used --source and dropped the invalid line
✓ spec-fix Fix a function to match its spec 78s follows the whole docstring
✓ feature Add a feature across files 86s priorities work, including old todos
Attempt 2 of 3
…
At the end it prints a table comparing the models (see results below).
A model usually takes 4 to 12 minutes per attempt at the quick tasks; each stops after 5 minutes,
and the longer tasks after 10 (--timeout sets one limit for all).
Three attempts at each task¶
Models sample their answers, so a task near the edge of a model's ability can pass in one run and
fail in the next. One run of each task is within that noise, so lcode bench makes three attempts
at every task, each in a new folder and conversation: every task once, then every task again. The
summary adds:
| pass@1 | The share of attempts that passed, averaged over the tasks: the chance that one attempt works |
| pass^3 | The share of tasks that passed all three attempts: the ones you can count on |
| Each attempt | How many tasks passed in each attempt. The difference between the best and the worst is the spread between runs |
They're counted as in the public benchmarks. --attempts 1 runs each task
once, as lcode bench did before 0.57, in a third of the time; --attempts 5 gives pass^5. In the
summary, a task's time is that of one attempt, on average, and Time is the total.
More tasks and features¶
--tasks find-concept adds a task in a larger repository: 40 files of a small backend where the
model has to find a function by what it does. It's there to measure --repo-map, which gives the
model the repository map, and --code-search, which indexes each task's folder for
semantic search. Both are off in lcode bench by default, so
results compare across machines and versions.
A second try, and a stronger model¶
--retry gives a failed task one more try, as a user would ask: "That didn't fix it: it still
doesn't work right. Look again." --escalate MODEL sets a stronger model
that's asked when the model is stuck and, with --retry, before the second try, as /escalate
would: no questions asked, since the flag is your OK. Comparing the two shows what the stronger
model's advice adds to the same second chance:
lcode bench --tasks runtime-bug --retry --attempts 3
lcode bench --tasks runtime-bug --retry --attempts 3 --escalate ollama/qwen3.8-27b
The JSON report has, for each task, first_passed (with --retry, whether the first try passed)
and escalations.
Long sessions¶
Each task normally starts a new conversation. --session runs them all in one conversation
instead, as a long working session would, so the context fills up and lcode's
context management has to work: the summary shows how often
it removed old tool output and how often it summarized the conversation. --no-prune turns the
removal off, to compare, and --compact-at sets the ceiling (--compact-at 0: only at 85%).
lcode bench --context 32k --session --rounds 3
lcode bench --context 32k --session --rounds 3 --no-prune
With --session, an attempt is one conversation, and --attempts is 1 unless you set it. --rounds 3
runs the tasks three times in it, 24 in one conversation, which fills a 32K window several
times over. With qwen3.6-35b on a 12 GB laptop (two runs each; runs vary a lot from one to the
next):
| Removing old tool output first | Summarizing only (--no-prune) |
|
|---|---|---|
| Conversation summarized | 1 and 2 times | 1 and 3 times |
| Old tool output removed | 3 times in each run | — |
| Tasks solved | 21 and 22 of 24 | 23 and 21 of 24 |
| Time | 26 and 28 min | 21 and 30 min |
Removing old tool output means fewer summaries, so more of the conversation stays word for word, at no measurable cost in tasks solved or time. The bench tasks are independent of each other, which is where summaries hurt least; in a real session, where later requests build on earlier ones, keeping the conversation intact matters more.
With a large window, compact_at decides when lcode makes room. In a 128K window, 24 tasks in one
conversation reach about 40K tokens, so the default ceiling of 64K never comes into play. Making
room earlier than needed doesn't cost tasks:
| 128K window, 24 tasks | No room made (peak ~40K) | --compact-at 32k |
|---|---|---|
| Conversation summarized | 0 times | 1 time (old tool output removed once) |
| Tasks solved | 21 of 24 | 21 of 24 |
| Time | 33 min | 27.5 min |
| Average prompt | 22.9K tokens | 16.6K tokens |
The tasks¶
Every task runs in a new temporary folder, with every permission granted (yolo mode) and web
access off, and ends with an automatic check. The folders are deleted afterwards unless you pass
--keep, which also keeps each task's transcript (lcode-bench.log).
Quick tasks¶
| Task | The model has to | Passes when |
|---|---|---|
fix-bug |
Find and fix a bug so the failing unit tests pass | The tests pass and the test file is unchanged |
find-code |
Answer a question about a small codebase | It names the right function and cites its file:line |
write-script |
Write a script from a spec and run it | The script prints the right output, also for data it never saw |
rename |
Rename a function across several files and tests | The old name is gone and extra hidden tests pass |
precise-edit |
Change one value in a 900-line file of near-identical entries | Exactly that one line changed |
recover |
Run a command that fails twice (a removed option, a bad input line) and work around it | The report is correct and only the bad line was dropped |
spec-fix |
Fix a function from a one-line bug report so it does everything its docstring says | Hidden tests of every rule in the docstring pass, edge cases included |
feature |
Add priorities to a small todo app (a new option, sorting, a marker in the output) | Its tests pass, and a scripted session gives the exact expected output, including for todos saved before the change |
The first six tasks are everyday work that good models get right; the last two need careful
reading and are where smaller models tend to slip. The checks are deterministic and can't be passed
by hard-coding: for example, write-script reruns the script on new data, and rename, spec-fix
and feature are checked with tests the model never saw.
Longer tasks¶
Good models now get every quick task right: qwen3.6-35b passed all 24 attempts. Four longer tasks, closer to real work, tell good models from great ones:
| Task | The model has to | Passes when |
|---|---|---|
cli-tool |
Write logsum.py, which summarizes web server logs, from a spec: options, text and JSON output, standard input, malformed lines, unreadable files and exit codes |
14 runs on logs it never saw give the same output and exit status as a reference implementation |
flaky-test |
Find why a test fails in about half the runs, with the same code, and fix it without changing the tests | The original tests pass under 24 hash seeds, and so does a hidden test of the order the code promises |
repo-bug |
Fix a bug reported only by its symptom (a second quote gets the first one's shipping cost) in a 47-file repository, keeping its lookups cached | Hidden tests of the report, of the same cause elsewhere and of the cache pass, and so do the repository's own tests |
wide-feature |
Add monthly budgets to an expense tracker: a new command, the report in text and JSON, and the data file, with older data files still working | Its tests pass, and a scripted session from an older data file gives the expected output |
fast-dedupe |
Make a job that merges duplicate customer records fast, from hours to seconds for 200,000 records, without changing its output | On 60,000 records it never saw, it finishes within a minute and its output is the old job's, byte for byte |
runtime-bug |
Fix a camera wall whose window grows taller with every video frame and can't be made smaller again. Reading the code suggests band-aids; running it shows the cause | A hidden test: the window keeps its size, can be made smaller, and the video fits its cell |
They're where qwen3.6-35b misses: see the results. In an earlier run
of the six it lost one fast-dedupe attempt to lcode itself: after lcode made room in the context,
the model lost track and repeated itself until the loop breaker stopped it.
Each rewards what a careful developer does: reading the whole spec, running a flaky test more than
once, fixing the cause rather than the symptom, trying old data, and comparing the new output with
the old. The longer tasks stop after 10 minutes instead of 5. --quick leaves them out and runs the
quick tasks once, as lcode bench did before 0.57.
What's measured¶
| Passed | Attempts that solved their task within the time limit; with several, also pass@1, pass^3 and each attempt's (above) |
| Time | Total time for the tasks, every attempt (loading the model is reported separately) |
| Generation | Output tokens per second, across all the model's responses |
| Prompt reading | Prompt tokens per second: how fast the model reads files and tool output. Measured per model with fresh 4,000-token prompts (the faster of two readings, since the first one after loading includes warm-up), because during the tasks Ollama serves most of each request from its prompt cache |
| Tool-call errors | Tool calls that failed (wrong arguments, editing without reading first, unmatched edits) or that the model wrote as invalid JSON |
| Memory | The model's memory and how much of it is on the GPU, counted as in the line after a load. Ollama leaves out what stays mapped from the model's file in system RAM, so for a model in lcode models that's partly off the GPU, it's lcode's estimate of the whole model (~23 GB). Otherwise it's what Ollama says |
Results depend on the context window, so lcode bench uses 32K by default on every machine,
which makes results comparable. When comparing models, each one is unloaded before the next starts
so every model gets the whole GPU; lcode warns if other models are already loaded. Use --context 128k to measure a larger window, and --no-think
to measure a model without reasoning.
Results on a 12 GB laptop¶
RTX 4080 Laptop GPU (12 GB) with 31 GB of RAM, Ollama 0.40.13, 32K context, reasoning on, lcode 0.58.0, three attempts at each task:
qwen3.6-35b |
|
|---|---|
| fix-bug | ✓✓✓ 25s |
| find-code | ✓✓✓ 14s |
| write-script | ✓✓✓ 15s |
| rename | ✓✓✓ 25s |
| precise-edit | ✓✓✓ 13s |
| recover | ✓✓✓ 23s |
| spec-fix | ✗✓✓ 1m 16s |
| feature | ✓✓✓ 1m 40s |
| cli-tool | ✓✓✗ 1m 23s |
| flaky-test | ✓✓✓ 39s |
| repo-bug | ✓✓✗ 2m 53s |
| wide-feature | ✓✓✓ 4m 17s |
| fast-dedupe | ✓✓✓ 1m 25s |
| runtime-bug | ✗✗✗ 5m 30s |
| Passed | 36/42 |
| pass@1 · pass^3 | 86% · 71% |
| Each attempt | 12/14, 13/14, 11/14 |
| Time | 62m 47s |
| Generation | 64.0 tok/s |
| Prompt reading | 851 tok/s |
| Tool-call errors | 4 |
| Memory | ~23 GB, 42% on GPU |
What it missed:
- one spec-fix attempt fixed the reported case but not all the docstring's rules;
- one cli-tool exited with 1 after skipping malformed lines, where the spec says 0;
- one repo-bug attempt went round in circles until lcode stopped it;
- every runtime-bug attempt settled for a band-aid: the window still grew, or still couldn't be
made smaller.
The table shows memory as lcode counts it now. lcode 0.58.0 printed Ollama's count,
10.5 GB, 98% on GPU, which leaves out the ~12 GB of experts in system RAM.
Two wide-feature attempts were first counted as failed. Their budget list printed No budgets
set. without budgets, and one listed the budgeted categories first. The task asks for neither
behavior, so the check now accepts both (0.59.0), and the table counts them as passed.
Earlier: the quick tasks, one attempt each¶
The quick tasks with three models, Ollama 0.32.15, 32K context, reasoning on, one attempt each (lcode 0.3):
qwen3.6-35b |
qwen3.5-9b |
qwen3.5-4b |
|
|---|---|---|---|
| fix-bug | ✓ 22s | ✓ 16s | ✓ 11s |
| find-code | ✓ 9s | ✓ 6s | ✓ 5s |
| write-script | ✓ 13s | ✓ 44s | ✗ 18s |
| rename | ✓ 26s | ✓ 28s | ✓ 30s |
| precise-edit | ✓ 10s | ✓ 12s | ✓ 8s |
| recover | ✓ 30s | ✗ 29s | ✓ 24s |
| spec-fix | ✓ 55s | ✓ 4m 40s | ✗ 2m 57s |
| feature | ✓ 1m 11s | ✗ 5m 00s | ✓ 47s |
| Passed | 8/8 | 6/8 | 6/8 |
| Time | 3m 57s | 11m 55s | 5m 20s |
| Generation | 71.4 tok/s | 60.1 tok/s | 91.3 tok/s |
| Prompt reading | 855 tok/s | 3,507 tok/s | 5,186 tok/s |
| Tool-call errors | 0 | 9 | 1 |
| Memory | 22.3 GB, 45% on GPU | 6.6 GB, 100% on GPU | 4.2 GB, 100% on GPU |
The small models fit entirely on the GPU and read prompts much faster, but the larger mixture-of-
experts model solves more and needs fewer attempts. The small models' failures were real mistakes:
copying read_file's line numbers into a data file, ignoring negative numbers, missing an edge case
of the spec, and running out of time.
Models sample their answers, so a task near the edge of a model's ability can pass in one run and
fail in the next (qwen3.6-35b scored 7/8 in an earlier run), which is why lcode bench now makes
three attempts at each task. Compare pass@1 and pass^3 rather than single tasks.
Sharing results¶
Results from other machines, especially Apple Silicon Macs and other GPUs, decide which models lcode recommends. Run
and paste the table into a model test report.
JSON format¶
--json FILE writes one document per run. The schema is versioned: fields may be added, but
existing fields keep their meaning until schema changes.
{
"schema": 1,
"lcode": "0.57.0",
"ollama": "0.40.13",
"server": "Ollama 0.40.13",
"date": "2026-11-02T09:44:14+01:00",
"hardware": {
"os": "linux", "cpu": "13th Gen Intel(R) Core(TM) i9-13980HX", "ram_gib": 30.96,
"gpu": "NVIDIA GeForce RTX 4080 Laptop GPU", "vram_gib": 11.99, "unified": false,
"description": "NVIDIA GeForce RTX 4080 Laptop GPU (12 GB VRAM), 31 GB RAM"
},
"tasks": [{"id": "fix-bug", "title": "Fix a bug so the tests pass"}, …],
"runs": [
{
"model": "qwen3.6-35b",
"ollama_model": "lcode-qwen3.6-35b",
"context": 32768,
"num_batch": null,
"think": true,
"load_seconds": 17.8,
"prompt_tps": 904.6,
"memory_gb": 10.4,
"gpu_percent": 98,
"estimated_memory_gib": 22.7,
"estimated_gpu_percent": 42,
"attempts": 3,
"passed": 24,
"total": 24,
"seconds": 756.2,
"generation_tps": 72.2,
"tool_errors": 2,
"summary": {"pass_at_1": 1.0, "pass^3": 1.0, "runs": [1.0, 1.0, 1.0], "spread": 0.0},
"usage": {"requests": 184, "prompt_tokens": 1058154, "prompt_ns": 199916607993,
"output_tokens": 38683, "output_ns": 535438114998},
"tasks": [
{"id": "fix-bug", "passed": true, "seconds": 21.5, "detail": "the tests pass",
"steps": 5, "tool_errors": 0, "error": null, "attempt": 1},
…
],
"error": null,
"interrupted": false
}
]
}
| Field | Meaning |
|---|---|
ollama, server |
The Ollama version, or null with another model server; server names the server either way |
runs[].model |
The name you gave (catalog key or Ollama tag); ollama_model is the model that ran |
runs[].context, num_batch, think |
The settings used. lcode lowers the batch size or context if the GPU runs out of memory, and reports what it ended up using |
runs[].memory_gb, gpu_percent |
The model's memory in GB and the share of it on the GPU, as Ollama's /api/ps counts them; null if unavailable. Ollama 0.40 leaves out what stays mapped from the model's file in system RAM: qwen3.6-35b at 32K on a 12 GB GPU shows as 10.4 GB, 98% of it on the GPU |
runs[].estimated_memory_gib, estimated_gpu_percent |
When Ollama's count leaves out part of a model in lcode models: lcode's estimate of the whole model in GiB, and the share of it on the GPU, as the summary shows them (~23 GB, 42% on GPU). null when Ollama counts the whole model (all of it on the GPU, or a model outside the catalog) and in older results |
runs[].generation_tps |
Output tokens per second across the tasks; null if Ollama reported no timings |
runs[].prompt_tps |
Prompt tokens per second for a fresh 4,000-token prompt (the faster of two readings); null if it couldn't be measured |
runs[].usage |
Totals over the tasks as Ollama reports them. prompt_tokens includes tokens served from the prompt cache |
runs[].attempts |
Attempts at each task (--attempts) |
runs[].summary |
pass_at_1, pass^3 (with the number of attempts), runs (each attempt's pass rate) and spread (the best minus the worst), counted as in the public benchmarks; null where there's nothing to count |
runs[].tasks[] |
One entry per attempt at a task; attempt says which (1, 2, 3) |
runs[].tasks[].steps |
Model responses in the task |
runs[].tasks[].error |
Why the task stopped early (timed out after 300s, an Ollama error), otherwise null |
runs[].error |
Why the model didn't run at all (for example, not installed), otherwise null |
runs[].interrupted |
You pressed Ctrl+C; the results cover the tasks that finished |