Qwen3.8-27B with 256k Context on a 3090

I’ve been running local models on my home server for a while now, mostly to work out where the line is for actually useful work — long-running agents, code generation, the sort of thing I’d otherwise be handing to somebody else’s GPUs. Qwen3.5 was good. Qwen3.8 promises to be better.

Which makes context length most of the game. An agent that has to be re-briefed every twenty minutes isn’t doing long-horizon anything, and code generation against a real codebase eats tokens fast.

So when Qwen3.8-27B landed this month, the thing I wanted to know immediately was whether I could run it at its full native context — 262,144 tokens — on the single RTX 3090 in that server. 24GB of VRAM, a 27B dense model, and a quarter million tokens of context is not an obviously winnable fight.

It fits. Barely, and only with the right KV cache configuration, but it fits: 22.5GB of 24GB used, 35 tokens/sec generation, 1237 tokens/sec prefill.

Here’s the arithmetic that gets you there.

Picking a harness

The model ships as GGUF from unsloth, which narrows the field. vLLM is the usual answer for serving, but it wants safetensors and gains you very little on a single Ampere card with no FP8 to speak of.

So: llama.cpp. One thing worth checking before you commit, though. Look at what the GGUF actually declares as its architecture:

general.architecture = qwen35
qwen35.block_count = 65
qwen35.full_attention_interval = 4
qwen35.attention.head_count = 24
qwen35.attention.head_count_kv = 4
qwen35.attention.key_length = 256
qwen35.attention.value_length = 256

Qwen3.8 is built on the Qwen3.5 architecture, so the arch string inside a Qwen3.8 file is qwen35. That’s the identifier that has to exist in llama.cpp’s llama-arch.cpp for any of this to work, and it does — LLM_ARCH_QWEN35, added a while back. If you go looking for “qwen38” support you will conclude, incorrectly, that nobody has implemented it yet.

The KV cache is the whole problem

Those hparams above are the ones that matter. Qwen3.8 is a hybrid: 64 layers laid out as 16 repetitions of 3 × Gated DeltaNet → 1 × Gated Attention. The DeltaNet layers are linear attention and carry a fixed-size recurrent state that doesn’t grow with context — a couple hundred MB, total, regardless of how long your conversation is.

Only the 16 full-attention layers keep a real KV cache. That sounds like a huge win, and it is, but those layers are not cheap: head_dim is 256, which is double what you’d expect, against 4 KV heads. Per token:

16 layers × 4 KV heads × (256 + 256) = 32,768 elements/token

Which lands as follows:

16 GiB of KV cache at f16 is a non-starter — that’s most of the card before you’ve loaded a single weight. Even q8_0 at 8.5 GiB leaves you choosing between context and a usable quant.

q4_0 at 4.5 GiB is what makes 262,144 work.

The quant

Working backwards from that: 24 GiB total, minus 4.5 GiB of KV cache, minus ~200MB of DeltaNet recurrent state, minus roughly a gigabyte of compute buffers, leaves about 17.5 GiB for weights.

UD-Q4_K_XL is 16.7 GiB. Unsloth’s dynamic quants keep the tensors that matter at higher precision, which is worth having when you’re already accepting q4_0 on the cache.

The repo also ships mmproj-F16.gguf, a working vision encoder (qwen3vl_merger, supported by llama.cpp’s mtmd). It costs about 0.9 GiB. I don’t need vision, so that 0.9 GiB went to the KV cache instead — which is most of the reason 262,144 fits comfortably rather than by a hair.

The config

Running under Docker, pinned to a specific llama.cpp build:

services:
  llamacpp:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda-b10454
    container_name: llamacpp
    restart: unless-stopped
    volumes:
      - /data/apps/llamacpp/models:/models:ro
    ports:
      - "8081:8080"
    command:
      - --model
      - /models/Qwen3.8-27B-UD-Q4_K_XL.gguf
      - --alias
      - qwen3.8-27b
      - --n-gpu-layers
      - "99"
      - --ctx-size
      - "262144"
      - -fa
      - "on"
      - --cache-type-k
      - q4_0
      - --cache-type-v
      - q4_0
      - --parallel
      - "1"
      - --jinja
      - --chat-template-kwargs
      - '{"reasoning_effort": "medium", "preserve_thinking": true}'
      - --reasoning-format
      - deepseek
      - --temp
      - "1.0"
      - --top-p
      - "0.95"
      - --top-k
      - "20"
      - --min-p
      - "0.0"
      - --host
      - 0.0.0.0
      - --port
      - "8080"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Flash attention is not optional here — quantized KV requires it. --parallel 1 matters more than usual on a hybrid model: every slot gets its own copy of the DeltaNet recurrent states, so extra slots cost you real VRAM on top of dividing your context window.

I pinned the image tag rather than tracking server-cuda, because I run watchtower and the qwen35 code path is new enough that I’d rather not discover a regression at 4am.

Things that look like problems and aren’t

On startup you get fifteen of these:

W model has unused tensor blk.64.attn_q.weight -- ignoring
W model has unused tensor blk.64.nextn.eh_proj.weight -- ignoring
...

Note block_count = 65 in the hparams against 64 layers of actual model. Block 64 is the multi-token prediction head. llama.cpp is telling you it skipped loading tensors it isn’t using — which is the correct behaviour, and better than the alternative, since those tensors would otherwise be occupying VRAM to do nothing.

I’d like to be using MTP, since it’s free throughput on a memory-bound workload. Currently native MTP allocates a second CUDA compute arena and OOMs (llama.cpp#27282), which on a card this full is fatal. Worth revisiting when it lands.

The other one to know about: the chat template defaults reasoning_effort to xhigh. At 35 tokens/sec, xhigh will think for a genuinely long time before it says anything. I default the server to medium and let clients override per request.

Results

prompt eval: 1237 tok/s   (18,620 token prompt)
generation:    35 tok/s
VRAM:      22,564 MiB / 24,576 MiB

Tool calling works out of the box with --jinja — llama.cpp routes the Qwen3-Coder-style XML parser, and calls come back properly structured. With --reasoning-format deepseek the thinking trace lands in message.reasoning_content instead of the body, which is what you want for anything agentic.

Pointing opencode at it is unremarkable:

{
  "provider": {
    "kraken": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "kraken (local)",
      "options": { "baseURL": "http://192.168.1.5:8081/v1" },
      "models": {
        "qwen3.8-27b": {
          "name": "Qwen 3.8 27B UD-Q4_K_XL",
          "limit": { "context": 262144, "output": 131072 },
          "options": { "temperature": 1.0, "topP": 0.95, "topK": 20 }
        }
      }
    }
  }
}

The tradeoff you’re actually making

262,144 tokens of q4_0 cache and 131,072 tokens of q8_0 cache cost almost exactly the same VRAM. I went wide because I wanted to see if it would work, but if long-context recall ever feels soft, halving the window to double the cache fidelity is a two line change.

And be realistic about what a quarter million tokens means at 1237 tok/s prefill: filling that window takes about three and a half minutes. The context is there when you need it, but it’s not somewhere you want to live.