2026-10-24 21:10:23 - No Comments
This is the "unscientific" edition: nine local models through the same 99 agentic coding tasks, 3 repeats each, on the opencode 1.18.16 harness — fixed 300s timeout, 3-repair budget. No cross-prompt averaging, no statistical treatment, and the boxes were under real load. The goal was never a league table; it was to find what this rig can actually run at the top end.
That ceiling just moved. Two quants of Qwen3.8-Flash-Next — a 125B Mixture-of-Experts with only a few billion params active per token — took the top two seats:
qwen3.8-flash-next-iq3_s(A6000, 48GB) — a clean 99/99 (100%) first-try, zero repairs.qwen3.8-flash-next-coder-iq1_m(3080 Ti, 12GB + 32GB RAM) — 97/99 (98.0%). A 125B MoE on a consumer card. That's the whole point of the post.
The catch is speed. Those two quants sit at 10.5 / 12.9 tok/s — bottom half of the group — because a 125B model doesn't decode fast on prosumer hardware. The older, smaller ones (nemotron 46.7, laguna 34.1, ornith 34.6 tok/s) run faster but land lower on pass-rate (73–92%). So the honest read is a trade, not a victory lap: when you need the model to actually get the job done right, Flash-Next wins — and you pay for it in wait time. For high-throughput single-shot, the smaller models still have the edge.
Treat the tok/s column with the same skepticism as everything else below.
Local model benchmark — harness: opencode 1.18.16
| Model | Pass rate | First-try validity | Avg repairs | Avg generation | Avg execution | Tokens/sec |
|---|---|---|---|---|---|---|
laguna-xs.2:latest |
91/99 (91.9%) | 100.0% | 0.00 | 93.31 s | 224 ms | 34.1 |
muse-glimmer:30b |
87/99 (87.9%) | 100.0% | 0.00 | 2m 11s | 172 ms | 11.6 |
nemotron-3.5-lightning:30b |
78/99 (78.8%) | 98.9% | 0.01 | 2m 45s | 124 ms | 46.7 |
ornith:35b |
72/99 (72.7%) | 100.0% | 0.00 | 49.96 s | 3.40 s | 34.6 |
qwen3.6:27b |
85/99 (85.9%) | 100.0% | 0.00 | 2m 18s | 129 ms | 3.9 |
qwen3.6:35b |
78/99 (78.8%) | 100.0% | 0.00 | 53.89 s | 3.43 s | 23.9 |
qwen3.8-flash-next-coder-iq1_m |
97/99 (98.0%) | 99.0% | 0.01 | 81.92 s | 158 ms | 10.5 |
qwen3.8-flash-next-iq3_s |
99/99 (100.0%) | 100.0% | 0.00 | 88.81 s | 173 ms | 12.9 |
qwen3.8:27b |
91/99 (91.9%) | 100.0% | 0.00 | 72.79 s | 129 ms | 15.4 |
Run dates 2026-08-13 to 2026-10-24 · repeats 3 · exec timeout 300 s · repair budget 3 · prompt 1.0.0
Be the first to leave a comment!