I moved to DwarfStar, and the bottleneck moved too

The short answer

I run a local model, Qwen3.8-Flash-Next, on a 64 GB Mac, and my assistant Hermes ↗ talks to it all day. The server is DwarfStar ↗, Salvatore Sanfilippo’s inference engine. Until this week I ran Ivan Fioravanti’s fork of it, as set up in part three ↗, because upstream did not carry this model. On 14 September it did, so I moved. Reading a prompt got 66 percent faster.

The next morning, replies from my assistant took minutes instead of seconds, and the server was as fast as ever.

The cause was a setting I had carried over unchanged. DwarfStar keeps one conversation in memory and writes the others to disk, so that it can bring them back when their turn comes. That morning three conversations took turns, and the 24 GB of disk I had allowed for those copies was not enough to hold them. The server kept deleting the copy it was about to need and re-reading the whole conversation instead, 50 to 60 thousand tokens at a time. Raising the disk budget to 64 GB fixed it: a switch between two 100K-token conversations now restores in about a second.

The rest of this note is the move itself, with the new server command; the arithmetic of why 24 GB was not enough and how to size it; and two things I tried or found upstream on the way: holding two conversations in memory at once, which has a fix waiting in PR 765 ↗, and a case where a saved copy sits on disk but is never found.

Moving from ds4-metal to DwarfStar

This note replaces the server half of the previous recipe ↗. Upstream is faster to read and to start, a little slower to write at mid-depth, and simpler to run: one self-contained 137 GiB model file, with the model’s 95 GiB lookup table inside it and read from the SSD on demand, instead of the fork’s separate 30 GB lookup file, which was a lower-precision copy of the same table. That removes two startup options the fork needed, the --ple flag and its eviction setting.

To move: clone antirez/ds4 ↗ at f06d3eb or later and build it with make; fetch the upstream model files with ./download_model.sh qwen38-q2 in the checkout, which brings the 137 GiB GGUF and the vision projector from antirez/qwen3.8-flash-next-gguf ↗; clear the checkpoint directory, because the two builds do not share a format; then start the server with:

cd ds4
./ds4-server \
  -m gguf/Qwen3.8-Flash-Next-Q2.gguf \
  --vision gguf/mmproj-Qwen3.8-Flash-Next-Q8_0.gguf --metal \
  -c 163840 --prefill-chunk 1024 --mtp \
  --kv-disk-dir ~/.ds4/server-kv --kv-disk-space-mb 65536 \
  -n 16384 --host 127.0.0.1 --port 8001

Compared with the fork’s line: no --ple, no DS4_QWEN4_PLE_EVICT_TOKENS, a new model file, and a disk cache budget of 64 GB instead of 24, for the reason this note is about. Hermes does not change at all: same endpoint, same model names, same config. Ivan’s model files can go once you are happy; mine were 72 GB.

Decode is about 14 percent slower on upstream at 60K, and from 80K to 160K it sits inside the fork’s range. For an agent that spends most of its time reading tool results, prefill is the number that matters, so I switched. The rare mid-reply hang from the previous note, where the exact-sampling option replays the whole context, is still there upstream: three stalls in fifteen turns with it on, none with --mtp alone, so the recipe still leaves --mtp-exact-sampling out.

On my machine the switch was that edit to the macOS service definition and a restart, on the morning of 15 September.

The morning it got slow

The time went to re-reading conversations and waiting in the queue, not to generating. Hermes’s own log records how long each reply took to produce:

The server, in the same two hours: prefill 486 to 522 tokens per second, decode 38 to 44. Identical to the day before.

06:45 to 07:41, three conversations. My main session was at 87K to 103K tokens. Two scheduled finance jobs were growing, one from 9K to 60K tokens, the other from 10K to 58K. The server holds one conversation live. Every time the next request came from a different conversation, the log said live kv cache miss, and the request queued behind whichever one was already re-reading. Calls in this window took 68 to 363 seconds each, including calls of mine that had 90K tokens cached and only had to wait. At 07:30 I pressed stop on a reply that had made thirteen calls.

07:46, compaction. The main session reached 126K tokens and Hermes compacted it, which means it summarised the older messages to shrink the session: 208 messages became 146, about 70K tokens. Compaction is one full prefill of what is kept, and it landed between the two windows.

07:55 to 08:19, two conversations in one session. After the compaction, my turns at 70K to 77K alternated with a background skill review that Hermes runs as a concurrent turn inside the same session, at 62K. Thirteen calls, two in flight at a time, most of them 200 to 266 seconds from request to reply, and every one reported only 14,336 cached tokens: the system prompt and nothing else. The skill system rejected every write the review attempted, so that time changed nothing.

Why the disk cache did not help

A conversation leaves far more on disk than its final copy, and that is what overflowed the budget. The previous note ↗’s cost table has a row for this, “switching between two long sessions: a full prefill, about 8 minutes at 160K”, and I wrote it believing the disk cache made that rare.

Three facts explain the morning.

1. One checkpoint costs 34.5 KB per token. The final copy of a 77K-token conversation is 2.6 GB.
2. A prefill writes a checkpoint every 10,240 tokens, and each one holds everything before it. In my log the first one appears at 20K, so reading a 77K conversation from scratch left seven files, 0.7 to 2.6 GB each, 11.6 GB in total, until the superseded ones are cleaned up. The 63K conversation it alternated with left another 8.9 GB the same way.
3. 8.6 GB of the previous day’s checkpoints were still on disk.

When the budget is full the server evicts the lowest-scoring checkpoint, and the score is mostly how often a file has been restored. A file the current prefill has just written has never been restored, so it scores lowest. Every one of the 434 checkpoints evicted in that 26-hour log had zero hits: the evictor kept deleting the newest file, the one the next switch was about to ask for. The only file that kept coming back was the 14,336-token system prefix, which every conversation shares. So each switch got the system prompt back and re-read the other 50 to 60K tokens at 510 tokens per second, 95 to 120 seconds, and then waited in the queue.

How to size it. The server does not document --kv-disk-space-mb, and the default is 4 GB. Because every checkpoint holds the whole prefix, the chain a cold prefill writes grows with the square of the conversation’s length: 11.6 GB measured at 77K, about 32 GB at 130K. So budget the chain of the longest conversation you might ever read from scratch, plus the latest copy of every other conversation that can be active the same day, 4.5 GB each at 130K. For me that is about 32 plus 3 × 4.5, under 50 GB, so 64 GB holds it. When the budget holds the whole chain, the superseded prefixes in it are the first thing the evictor drops when it needs room, which is the right order; it was only when the chains did not fit that it reached the files still in use. If disk is short, --kv-cache-continued-interval-tokens sets how often the chain is written; doubling it to 20480 halves the chain at the price of re-reading up to 20K tokens after a crash. Both the quadratic chain and the eviction order under a full budget are issue 444 ↗, open since June with a proposal to store checkpoints as a tree of chunks.

The fix

If you run this setup, the change is one flag on the server:

--kv-disk-dir ~/.ds4/server-kv --kv-disk-space-mb 65536

Size it as above: the chain of your longest cold prefill, about 32 GB at 130K, plus 4.5 GB for every other conversation that can be active the same day. Clear the directory when you change models or builds; the checkpoint format is not the same across them.

I made the change at 09:46 on 16 September, forty seconds of downtime, and tested it the next day with two long sessions alternating for eight turns: every switch restored from disk in about a second, each turn cost only its own new tokens, and nothing was evicted. The turn-by-turn numbers are in the details, with the three-conversation case and the one bug it turned up.

What two resident sessions do and do not fix

Two resident slots only help once both are occupied, unless you run the fix that is waiting upstream, and on this machine they cost memory and, under load, your reply’s speed. While I was diagnosing this, upstream gained --batched-session N: N conversations held in memory at once, with the weight-reading work batched across them and the caches kept per session. It should remove the switch entirely. I built the latest upstream and tested it the same way.

Sending A, then B, then A never uses the second slot. The scheduler scores each idle slot by how much of the incoming request’s prefix it already holds and picks the highest. An empty slot holds nothing and scores zero. The slot holding A shares the test harness’s 11K system prompt with B, Hermes’s own is the 14K one, and scores 11,154, so B goes into A’s slot and pushes A out to disk; A comes back and pushes B out. The second slot is only ever filled when two requests overlap in time, and from then on both conversations stay resident and the scorer routes each correctly. I reproduced it at 160K and at 128K and reported it as issue 1069 ↗. It turned out that kmike had found the same thing in August, with sub-agents as the trigger, and has a fix waiting in PR 765 ↗ that routes a request to a slot whose cache it can actually reuse and takes empty slots first. I built that branch and ran the same two sessions through it: B went to the empty slot, A came back to its own, and across nine turns the server logged no live miss, no disk restore and no eviction. That is the row with the PR in the table above, and it is the one to want.

The concurrent slowdown is by design. A prefill yields to decoding sessions once every 128 tokens, and at 500 tokens per second that is about four times a second, so a session that is writing its reply gets only a few tokens a second, 2.3 to 3.3 in my runs, instead of forty-five. When a cron job starts reading a long page while you are waiting for a reply, your reply crawls. With one slot the cron job waits for your reply to finish; with two, its reading slows your reply down.

Memory is the other constraint. Two 160K slots reached 56.3 GiB wired, which is the most this 64 GB machine lets the GPU pin, with 5 percent free. Two 128K slots, both holding 120K to 130K tokens, peaked at 54.2 GiB with 10 percent free and no growth in swap, 3.6 GiB more than the same test with one slot, which peaked at 50.6 GiB. So two slots fit at 128K, with less headroom than I run with today. The startup line still printed the one-slot plan of 48 GiB, because the second slot’s caches are allocated when first used, so a server that starts is no evidence that it fits.

My daily server stays at one slot. The disk cache is the fix for switching. Batching is for people who need two conversations to make progress at the same time and have the memory for it.

What to expect

Known limits

• The restart-restore miss, issue 1053 ↗, also reproduces with DeepSeek models on the same engine, so it is not specific to Qwen. It has an open fix, PR 1058 ↗; two people found a second gap in its smoke test, so it is not merged yet.
• The disk cache has no sizing guidance, and the default budget of 4 GB is one checkpoint of a 120K conversation. Size it, and clear it when you change models: the checkpoint format differs between the fork and upstream. The eviction order that bit me is issue 444 ↗.
• Batched sessions: cold-start assignment, issue 1069 ↗, fixed by PR 765 ↗ once it merges, and the decode starvation during a neighbour’s prefill.
• A tool conversation whose last reply was plain text is checkpointed under a different key from one whose last reply was a tool call, and the next request does not find it. In the overnight three-conversation test below, that conversation re-read 110K to 131K tokens on every turn with its checkpoint sitting on disk. It is the in-process form of issue 1053 ↗, and PR 1058 ↗ is the fix to watch.
• Besides the cache budget, Hermes’s compaction threshold is a setting I have not tuned. It is set at 143K of a 160K window, and Hermes’s preflight check fires at 123K, so the session reached 126K before compacting, which makes every cost above bigger. At 49K, on the earlier model, a long task compacted four times in an hour and lost file text each time; I have not found the right number in between.

How the numbers were taken

The server log is timings, token counts and cache decisions, but since 15 September the same server also serves a finance agent, and a failed tool-call parse can log a few hundred characters of model output. Every number in this note was extracted with pipelines that print only fields after ctx=, chunk=, avg=, tokens= and the like, never a raw line. The Hermes side came the same way from its API-call lines, which carry prompt size, cached tokens and latency. The pipelines printed nothing else.

The test harness

session_bench.py, from the previous note, grows a Hermes-shaped session by one tool call and one 10K-token tool result per turn, resends the history the way Hermes does, and records prompt size, cached tokens, tokens re-prefilled according to the server log, and time to first token. For the alternation tests I saved two sessions and replayed them alternately, one turn at a time. The drivers pause Hermes’s cron jobs, stop the daily service, run the phases, and restore the service with a hard stop, then check that launchd owns the process answering on the port.

The switching test

Two tool-heavy sessions of 70K and 80K tokens, grown by a 10K tool result per turn, alternating for eight turns on the daily server with the cleared 64 GB cache, on 17 September.

Every switch restored the whole conversation from disk in about a second, and each turn cost only its own new 10K tokens, 23 to 26 seconds. Zero evictions, with the sessions at 110K and 120K by the end. The day before, the calls in the review window took 200 to 266 seconds, queueing included.

Three conversations, overnight

The control test had two conversations. The morning had three, so on the night of 17 September I ran three on the daily server with the 64 GB cache, no restart: A and B were the switching-test sessions, one turn on at 120K and 130K, C was built fresh to 100K, then three rounds of A, B, C, each turn adding a 10K tool result.

Two findings. The cache did fill: 64 GB, 28 files, 67 evictions. Sixty-five of the victims were superseded chain prefixes with zero hits, which is what the evictor should drop first, and neither A nor B ever lost the checkpoint it needed. The sizing rule above comes from this run.

C’s misses are not evictions. Its checkpoint was written after every turn and never evicted; the log shows it stored under the plain-text key while A’s and B’s were stored and found under the tool-conversation key. The only difference in the conversations is that C’s replies contained no tool calls. That is the keying gap behind issue 1053 ↗, seen here without a restart, and it means a long conversation can pay a full re-read on every turn while its saved copy sits on disk. Until PR 1058 ↗ lands, the daily setup does not hit it, because Hermes replies with tool calls in every long session I have.

What went wrong

Two mistakes of my own, as in the last note. The first driver waited for “all background jobs” after a concurrent phase, which included my wired-memory sampler and the driver’s two-hour timeout, so it sat with the test server on the GPU for half an hour while Hermes was down. When I killed it, its restore ran twice, and the second pass’s cleanup rule, “kill any test server under the ds4-metal directory”, also matched the daily service, which now lives in ds4-metal-upstream. Launchd relaunched it in ten seconds. Both drivers now wait on explicit process ids, restore once, and never kill the process launchd owns. The second run had no memory sampler at all; the 128K figure above comes from a third, two-minute run with two samplers.

Who did what: I set the questions and made the calls, including which fixes to apply and which to leave alone. Claude, in Claude Code on the same machine, read the logs and the engine source, wrote the drivers, ran the tests, and drafted this note and the issue. The mistakes are its as much as mine.