2026-11-13 07:11:46 - No Comments
Local model benchmark — harness: opencode 1.18.16
This is the 236-task round: one pass each, same 300s exec timeout, 3-repair budget. Two effects show up at this scale that the 99-task run hid.The timeout is a first-class competitor. All three dense 27B rows average ~4½ minutes of generation — right against the 5-minute wall — at under 2 tok/s. On hard tasks they hit the cap and lose runs they'd win with time. The MoE rows finish in 25–82s and never see the ceiling.But speed alone doesn't buy it. MiniCPM-V-4.7 is second-fastest here and near the bottom (19.5%). LFM2.5 is fastest (93 tok/s) and worst (18.2%). What carries the table is active capacity: qwen3.6:35b tops it at 59.7% without ever touching the timeout.
Arete-27B vs its base. 86/236 against qwen3.8:27b's 101/236 — 15 tasks below the model it was built from. At one repeat, binomial noise at this pass rate is ±7–8 tasks per model, so a 15-task gap is roughly 1.4× the error of the difference: suggestive, not conclusive.
Avg repairs 0.00 across the board, which means every failure is a first-try failure — a capability gap or a timeout, not a repair loop. And treat the tok/s column with the same skepticism as everything above.
| Model | Pass rate | First-try validity | Avg repairs | Avg generation | Avg execution | Tokens/sec |
|---|---|---|---|---|---|---|
hf.co/bartowski/NousResearch_Arete-27B-GGUF:Q4_K_M |
86/236 (36.4%) | 100.0% | 0.00 | 4m 37s | 3.21 s | 1.0 |
hf.co/openbmb/MiniCPM-V-4.7-35B-A3B-gguf:Q4_K_M |
46/236 (19.5%) | 100.0% | 0.00 | 24.93 s | 14.09 s | 50.7 |
lfm2.5:24b |
43/236 (18.2%) | 100.0% | 0.00 | 77.81 s | 151 ms | 93.1 |
qwen3.6:27b |
94/236 (39.8%) | 100.0% | 0.00 | 4m 39s | 183 ms | 0.7 |
qwen3.6:35b |
141/236 (59.7%) | 99.5% | 0.00 | 81.75 s | 2.71 s | 35.9 |
qwen3.8:27b |
101/236 (42.8%) | 100.0% | 0.00 | 4m 30s | 7.28 s | 1.5 |
Run dates 2026-11-10 to 2026-11-12 · repeats 1 · exec timeout 300 s · repair budget 3 · prompt 1.0.0
Be the first to leave a comment!