Tuning Qwen3.8-27B for Hermes on a 64GB M4 Max

On a 64GB M4 Max, the setup that holds up for me is the regular Qwen3.8-27B MLX 4-bit checkpoint with DFlash2, served through oMLX 0.6.4.

Main model

Qwen3.8-27B MLX 4-bit ↗

Drafter

Qwen3.8-27B-DFlash2 ↗

Drafter quantization

4-bit weights / 16-bit activations

Hermes context

65,536 tokens

DFlash ceiling

98,304 tokens

Tool-result pruning

32,768 tokens

Compression

49,152 tokens; thinking disabled

Prefix cache

Four entries / 8GiB

None of the pieces does the job alone. Qwen3.8-27B carries the quality, DFlash2 keeps sustained generation fast, and the context and compression settings stop long Hermes sessions from degrading.

This was less a choice than a retreat. My earlier Qwen3.6 setup repeatedly ran into memory problems on this Mac, so I moved to cloud-hosted DeepSeek 4 Flash, then Pro, before returning to local inference with Qwen3.8-27B.

The MLX Community 4-bit conversion occupies about 15GB on disk. Without speculative decoding, I usually saw 23–25 tok/s. More importantly, it handled sustained Hermes work well: tool use was coherent and long tasks were more likely to finish.

I considered 8-bit and kept 4-bit. On a 64GB unified-memory machine, spare memory is useful for long prefills, KV state, the drafter and occasional request overlap. The existing quality made that headroom more valuable than heavier weights.

The names are easy to mix up. DFlash is the original speculative-decoding method: a small diffusion-style model drafts several future tokens in parallel, then the full model verifies them. DFlash2 is the newer drafter used here. It predicts a candidate block and selects a coherent path for Qwen3.8-27B to verify.

The DFlash2 checkpoint is the drafter; Qwen3.8-27B remains the answer model. Accepted output still passes through the target, preserving its greedy output and sampling distribution.

01

DFlash2 drafts

The small drafter proposes several future tokens at once.

↓

02

Qwen3.8-27B checks

The answer model verifies the proposed block.

↓

03

Generation skips ahead

The verified prefix is accepted; a rejected tail is tried again.

oMLX 0.6.2 lacked the complete runtime path. I tested the 0.6.3 release candidate, then moved to stable 0.6.3 when it added end-to-end DFlash2 support, sampling parity and prefix-cache compatibility.

Five representative long-context Hermes turns decoded at 24.9, 27.7, 26.3, 28.7 and 24.9 tok/s—a median of 26.3 tok/s. Real agent traffic includes short replies, tools and changing prompts, so it rarely resembles a clean long-generation benchmark. The gap between 44.6 and 26.3 matters: the headline number is a best case, not the experience.

Prefix reuse changed the feel even more. After a cache reset, a 28K prompt spent about 121 seconds in prefill. The next turn reused 28,169 tokens, reduced prefill to roughly 4 seconds and completed in about 16 seconds.

Hermes initially detected Qwen3.8-27B's advertised 262K context. The architecture supports it, while a 64GB machine needs a more practical everyday target.

To test what this Mac could use reliably, I built three long prompts of roughly 48K, 65K and 96K tokens. Each prompt placed five specific facts at different positions. After the prefill, I asked Qwen3.8-27B to return all five facts.

Qwen3.8-27B returned all five facts correctly at every tested length. The 96K cold request took about 557 seconds; a cached follow-up reused 96,093 tokens and completed in 15.27 seconds.

I use 65,536 tokens for normal Hermes conversations and 98,304 tokens as the largest context still eligible for DFlash. The second value gives unusual requests some room without making 96K the default.

Web research fills context quickly because search results and tool output enter the history. The current policy prunes old tool results at 32,768 tokens and begins compression at 49,152, leaving about 16K tokens of working room.

Thinking remains available for the main assistant. Compression and session-title generation use the faster non-thinking path.

Title generation needed a separate cap, and this one was embarrassing. One request accidentally generated 8,383 tokens and occupied the model for about 125 seconds. Setting max_tokens: 64 reduced the replay to 9 tokens in 2.10 seconds.

I also avoid overlapping large prefills. In one concurrency test, two simultaneous completions fell to 9.1 and 2.8 tok/s. Continuous batching helps, though shared GPU bandwidth and transient memory still make heavy parallel work expensive here.

Qwen3.8-27B includes native vision. In a simple check it correctly read three coloured shapes and the code HERMES-384.

DFlash2 accelerates text. When an image arrives, oMLX moves that model instance to its normal VLM engine and currently stays there for later text requests. I reproduced the behavior on 0.6.4; it matches oMLX issue #1598 ↗.

I do not manually reload the model after every image. There is no Hermes-side recovery step in this setup: the loaded instance remains on VLM until oMLX unloads it, for example when another model replaces it or the app restarts. The next text request then loads Qwen3.8-27B with DFlash2 again. If I need the speed back immediately, the direct recovery is to unload the model in oMLX and send another request. The log entry DFlashEngine loaded confirms that speculative decoding is active.

I tested the community Qwen3.8-Flash-Next-REAP-288-MLX-4bit conversion. It occupied about 69GiB on disk; SSD-assisted mapping reduced its loaded footprint to roughly 39–40GB.

The model passed small instruction, recall, tool-selection and planning checks with medium thinking. Near 60K, oMLX rejected prompts at 45,056 tokens on one pass and 49,152 on another. I capped it at 40,960 tokens, below the 65K target for Hermes.

This build keeps 288 of the original 512 experts, its pruning calibration focused on agentic coding, and vision after pruning remains untested. Its companion MTP drafter also reports roughly break-even acceleration on M4-class hardware.

For future variants, I have two hard gates:

1. Verified 65K or larger context on a 64GB Mac under oMLX.
2. Resident and peak-prefill memory figures.

Beyond those, I would want general-domain and tool-use evaluations alongside coding scores, confirmed vision quality after pruning, measured M4 acceleration from Lightning MTP or another drafter, and reliable prefix-cache restoration after SSD streaming.

model:
  default: Qwen3.8-27B-4bit
  provider: custom:local-omlx
  context_length: 65536
  supports_vision: true

compression:
  threshold_tokens: 49152
  proactive_prune_tokens: 32768

auxiliary:
  compression:
    model: Qwen3.8-27B-4bit
    timeout: 120
    extra_body:
      chat_template_kwargs:
        enable_thinking: false
  title_generation:
    model: Qwen3.8-27B-4bit
    timeout: 30
    extra_body:
      max_tokens: 64
      chat_template_kwargs:
        enable_thinking: false

On the same machine profile, I would start here and change one variable at a time. The useful numbers to watch are cold-prefill time, cached-prefill time, draft acceptance, peak memory and completion quality over a real multi-step task.