A 2-bit 125B model became my daily assistant on a 64GB M4 Max

The short answer

Qwen3.8-Flash-Next ↗ is a 125B mixture-of-experts model with 6B active parameters, so it needs the memory of a big model but generates at the speed of a small one. Ivan Fioravanti’s 2-bit quant ↗ of it, served by DwarfStar ↗, Salvatore Sanfilippo’s inference engine, through Ivan’s ds4-metal ↗ fork that adds this model, now runs my Hermes ↗ assistant, an open-source agent that answers my chats and runs my scheduled jobs, on a 64GB M4 Max. It replaced the dense 27B from my earlier notes ↗.

Three things had to be true for that to be a good trade, and I measured each: the quant still reasons like the full model, a normal turn stays fast as the conversation grows, and Hermes and the server agree on how to talk to each other. The first two held on their own. The third took seven settings and one bug fix, and that is the recipe below.

The recipe

You need a 64GB Apple Silicon machine, Ivan’s model files ↗, and a build of Ivan’s ds4-metal ↗ fork of DwarfStar, branch qwen3.8-flash-next, at ccea768 or later; upstream DwarfStar did not carry this model when I wrote this. At the 160K window used below the server plans 48 GiB of memory, so it cannot share the machine with another large model; if you run something like oMLX for another model, it has to be unloaded first.

1. Start the server

Run it as a launchd service, so that a shutdown is graceful and the live conversation is kept on disk:

cd ds4-metal
DS4_QWEN4_PLE_EVICT_TOKENS=1024 ./ds4-server \
  -m gguf/Qwen3.8-Flash-Next-IQ2XXSImatrix-Q2KDownPad768-MTP.gguf \
  --ple gguf/Qwen3.8-Flash-Next-PLE-Q4_1.gguf \
  --vision gguf/mmproj-Qwen3.8-Flash-Next-Q8_0.gguf --metal \
  -c 163840 --prefill-chunk 1024 --mtp \
  --kv-disk-dir ~/.ds4/server-kv --kv-disk-space-mb 24576 \
  -n 16384 --host 127.0.0.1 --port 8001

What the flags do, in plain words:

• -c 163840 is a 160K window. 224K fits too, and the 64GB thread ↗ shows how, but it leaves the machine at 7 percent free memory. I could not name a daily task that needs the last 64K.
• --prefill-chunk 1024 and the PLE_EVICT_TOKENS variable are the two 64GB knobs from Ivan’s docs. One shrinks the prefill buffers from 9 GiB to about 1, the other keeps the model’s 30 GB lookup table, the PLE sidecar, from slowly filling RAM as it pages in from disk.
• --mtp turns on speculative decoding, worth about 20 percent on decode. Leave --mtp-exact-sampling out: with it on, roughly one turn in ten replays the entire context before answering, minutes at 100K. The trace is in discussion #5 ↗.
• --vision loads the image projector so Hermes can send photos. Builds before ccea768 can crash on the first real image; that was discussion #6 ↗.
• --kv-disk-dir keeps checkpoints on disk. The conversation that is live when the server stops comes back in under a second after a restart.

2. Point Hermes at it

Add the server as a custom OpenAI-compatible provider at http://127.0.0.1:8001/v1 with model qwen3.8-flash-next, then set these in ~/.hermes/config.yaml:

model:
  default: qwen3.8-flash-next
  provider: custom:local-ds4
  context_length: 163840
  reasoning_echo: true          # the one people miss
auxiliary:
  title_generation:
    enabled: false
compression:
  threshold_tokens: 143360
  proactive_prune_tokens: 98304
delegation:
  child_timeout_seconds: 1800
providers:
  custom:local-ds4:
    stale_timeout_seconds: 900

Then pin every cron job to the new provider with hermes cron edit <id> --provider custom:local-ds4 --model qwen3.8-flash-next; Hermes deliberately skips unpinned jobs after the global model changes.

Those lines fall into three groups.

Keep the conversation cacheable. The server remembers the conversation it is in the middle of and, on each new message, reads only the part it has not seen. That works only while the history Hermes resends is exactly what the model produced, thinking included. Hermes strips the thinking by default, so without reasoning_echo the history stops matching at the first assistant turn and every message re-reads the whole conversation: 70 seconds at 24K, and growing. With it on, a turn at 28K reused all but 18 tokens and finished in 3.2 seconds. Title generation is off for the reason the picture shows.

Size the window. Compaction at 143K and pruning of old tool outputs at 98K, instead of the 49K and 32K that suited a 64K window. With the old values a long job pruned itself mid-run and paid a 73-second re-prefill for nothing.

Size the timeouts for local speed. Hermes’ defaults assume a cloud API. A research sub-agent at 45 tok/s needs about 20 minutes, not the default 10; a cold re-prefill of 56K tokens takes 177 seconds, which is just over the default 180-second cap on non-streaming calls and cost me a finance job the first morning. 1800 and 900 seconds cover the window.

Why it works

The quant holds. On IFBench ↗, thinking on at the card’s default effort and sampling, all 300 prompts, the quant scored 80.7 strict prompt-level against 81.3 on the bf16 model card, once the 31 prompts that ran out of a 30K token budget got a 60K retry. On the 269 prompts that finished within budget it scored 82.9, though those are by construction the ones it found easier. Five needle facts injected into a real 222K agent transcript were retrieved 5 of 5 at 10K, 59K, 118K and 190K, and 38 of 40 forced tool calls were made correctly. The one weakness is rumination: on letter-level constraints it can think for 30K tokens without committing. That looks like a budget question rather than a weights problem, but I could not check whether the bf16 model deliberates the same way, because Qwen does not publish the budget behind its number.

Prefix reuse holds at depth. Measured on a Hermes-shaped session growing by 10K-token tool results, every turn from 10K to 222K cost 27 to 37 seconds of prefill, about 3 seconds per thousand new tokens, and decode ran at 43 to 46 tok/s. Without reuse, a turn at 100K would take five minutes, which is why the Hermes half of the recipe is all about keeping it intact.

What to expect

Two real jobs ran on the trial day without an error: the finance pipeline, 48 requests and 73K tokens of prefill, and a research session with a delegated sub-agent that grew to 89K. When the main session took the slot back it restored from disk in a quarter of a second. Memory stayed inside the plan throughout: swap flat at 1.6 GB, wired memory 45 GiB idle and 52 GiB busy. The scheduled finance run the next morning is the one the 180-second cap failed, before that timeout was raised.

At rest the server holds 45 GiB wired and macOS reports about a fifth of memory free. A Next.js dev server and a few browser tabs fit; a large Xcode build does not.

Known limits

• The exact-sampling stall is open upstream; the workaround is in the recipe.
• Only the conversation that was live at shutdown survives a restart. Mid-session checkpoints mostly fail to save at this memory pressure, so a crash costs a full prefill.
• Reasoning stays in the context once echoed, so sessions grow faster than with the 27B. Compaction handles it.
• With exact sampling off, the speculative decoder accepts its guesses a little more readily than pure sampling would. That is a small bias in the output that I have not measured.
• 224K works, and was measured end to end, but I do not have a daily task that needs the last 64K and the memory headroom is worth more.

How the quant was benchmarked

The model card ↗ reports twelve text benchmarks. Nine are agentic harnesses that need Claude Code style scaffolding and hours per task. Two are reproducible on one machine: IFBench and GPQA Diamond. I ran IFBench in full with AI2’s own verifier, thinking on at xhigh, temperature 1.0, top_p 0.95, top_k 20.

Thirty-one prompts ran out of budget at 30K and scored zero; on the 269 that finished the score was 82.9. Given a 60K retry, 29 of the 31 answered and 19 passed. Only 10 of them actually needed more than 30K tokens; the other 19 finished shorter on a fresh sample, so the variance is in how long the model deliberates, not in what it knows. Qwen does not publish the budget behind 81.3, so the honest label for the second number is “30K with one retry at 60K”. GPQA Diamond was stopped at 14 questions, 10 right, 3 out of budget, 1 wrong; too few to quote and 15 to 20 hours of GPU for a number about graduate chemistry. Quesma’s study of the 27B ↗ is the closest published comparison for 2-bit behaviour.

The context ladder

A reply on Ivan’s announcement argued that memory capacity is not the interesting limit for a 2-bit model; fidelity past 128K is. I grew a Hermes-shaped session to 222K tokens, cut it at four depths, injected five needle facts at spread positions, and appended tasks as new turns: retrieve each needle, ten forced tool calls graded for correct tool, valid JSON, required fields, types and enums, and ten IFBench prompts.

The two tool-call misses answered in prose instead of calling the tool; neither was malformed JSON. The IFBench leg came out flat at five or six of ten at every depth and says nothing about depth: the Hermes persona says “be concise” and the model obeyed.

The stall, traced

Symptom: after a normal incremental prefill, the first token takes 31 seconds at 12K, 187 at 61K, 285 at 79K, 457 at 150K, always the context divided by the prefill rate, then decode runs at full speed. The log gives the mechanism away: the PLE sidecar prints one eviction line per 1,024 tokens gathered, and a stall shows exactly context ÷ 1,024 of them. The engine is replaying the entire transcript.

The table counts my runs on the earlier build. A separate 16-turn run on Ivan’s ccea768 build, with two lines of logging added, stalled six times, and every stall had the same shape: the reply’s last speculative block rejects its draft, the server rewinds one token, the one-token snapshot is at that exact position but already marked invalid, the block-start snapshot is valid but one position short, and the graph resets. The replay is paid at the next request’s first decode step, which is why it looks like a stall on the following turn. With --mtp alone the same rewinds all restore. The full trace and two candidate fixes are in discussion #5 ↗. One of those six stalls, with the clock running:

The image crash

The first photo I sent segfaulted the server in the image cache. The cache copied embedding rows 4,096 wide; the Qwen encoder emits rows at its projector width, 2,560, so every insert read 60 percent past the end of its buffer and crashed only when the overrun crossed a page boundary. A three-line fix, asking the engine for its width, held across twelve image requests at six sizes, and Ivan’s ccea768 push carries the same fix. Details in discussion #6 ↗.

Memory and the SSD

At 224K the server plans 50.3 GiB against a 56 GiB wired limit; while active, wired memory sits at 52 GiB and free memory at 7 to 8 percent. The kernel copes by evicting model and sidecar pages and re-reading them, about 85 MB/s while a turn runs. Over a 12-hour soak that was 477 GB read and 1.7 GB written, so the SSD is not a concern; I initially misread iostat’s combined column as writes. At 160K the KV plan drops to 5.2 GiB and the idle footprint to 45 GiB.

Two checkpoint findings: mid-session “continued” checkpoints mostly fail to save under this pressure, so only the live session survives a restart, and a full disk cache silently discards the save made when a session is evicted. Clear old checkpoints when switching models.

Timeline, and what went wrong

The work took a weekend and a Monday morning. Three of my own mistakes shaped the numbers and are worth knowing if you repeat this. I left a test server holding the GPU for eleven hours after a run finished, and two Hermes cron jobs failed because the 27B could not load beside it; since then no GPU test runs outside a script with a hard stop, with Hermes’ cron paused. My first turn harness stored an empty reply when the model answered with a tool call, which broke prefix reuse and made the server look forty seconds slower per turn than it is; the harness must echo exactly what a client echoes. And the 180-second non-streaming cap in Hermes failed a finance job on the first real morning before I found it.

Who did what: I set the questions and made the calls. Claude, in Claude Code on the same machine, wrote the harnesses, read the engine source to find both bugs, drafted the reports I posted, and ran the machine overnight. The mistakes are its as much as mine.