swap-stack: ornith duo 128K via mixed KV (q8 K + turbo2 V); turbo-KV root cause

Probe matrix proved K is the broken side of turbo KV on Qwen3-4B
(QK-norm gamma outliers vs PolarQuant's no-per-channel-range format)
while V tolerates 2-bit for free. Applied to ornith duo main: K q8_0 +
V turbo2 = 128K ctx in ~880MB RAM at full speed (8.7-10.1 t/s concurrent,
vs 3x decode loss with turbo2 K). Qwen duo stays q4/q4 24K — only clean
mix is bigger than q4/q4. InnerQ equalizer confirmed broken as shipped
(compensation error grows with scale strength); documented in
MOE-FINDINGS with the full matrix and fix ladder.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs/MOE-FINDINGS.md

@@ -65,15 +65,54 @@ ngl 99 owns the GPU (3240 MiB incl. KV+compute), ornith runs pure CPU.
 Net: subagent 5.8 → 43 t/s (7.4×) under concurrent load; main pays −25%
 (10.6 → 8.0). Subagent is now contention-immune (GPU decode, 3 CPU threads).
 
-**Context sizes (2026-07-10, verified loaded + benched):** ornith 64K, qwen 24K.
-Throughput unchanged (ornith 9.2 / qwen 43.2 concurrent).
-- Ornith is GDN-hybrid: only 10/40 layers carry KV → 64K q8 = just 680 MB, in
-  RAM (CPU model). 128K would work RAM-wise (+680 MB) but steals page cache
-  from the 13.1 GB GGUF — thrash risk, not attempted. Trained ctx 262K.
+**Context sizes (2026-07-10, verified loaded + benched):** ornith **128K** (q8 K +
+turbo2 V), qwen 24K. Concurrent: ornith 8.7-10.1 / qwen 43-44.5 t/s.
+- Ornith is GDN-hybrid: only 10/40 layers carry KV. Mixed KV types: K q8_0 +
+  V turbo2 = ~880 MB @ 128K in RAM. Full-turbo2 gated clean on this arch
+  (PPL ladder @8K: f16 6.813 / q8 6.838 / turbo2 7.009, Δ0.196 < 0.5) but
+  costs 3× CPU decode on the K side (9.2 → 2.2-3.3 t/s) — K stays q8.
+  V-side turbo2 is speed-free on CPU (probe-verified on qwen, confirmed here).
+  Trained ctx 262K; 128K KV no longer steals page cache.
 - Qwen3-4B is full-GQA: 40 KB/token even at q4_0 → KV 648 MB @ 16K, 1296 MB
   @ 32K = **OOM** (weights 2.3 GB + compute leave no room). 24K fits at
-  3564 MiB / 4096. turbo2/3/4 KV would halve it but is BROKEN on this model
-  (PPL 438 @ 32K — FINDINGS §2). 24K is the hard ceiling on 4 GB VRAM.
+  3564 MiB / 4096. turbo KV is broken on this model — see §Turbo-KV below.
+
+## Turbo-KV on Qwen3-4B: root cause + fix status (2026-07-10)
+
+Full investigation: 3 agents + empirical matrix. Fork source:
+github.com/TheTom/llama-cpp-turboquant, local clone ~/Sources/llama-cpp-turboquant
+(branch fix/innerq-clamp), patched images `*-innerq` built.
+
+**Root cause (confirmed):** turbo2/3/4 = PolarQuant per-128 head vector (one fp16
+norm + fixed WHT + fixed Lloyd-Max centroids, no per-channel range). Qwen3's
+QK-norm gamma has extreme per-channel outliers (blk.0 ch51 γ=44 vs mean 1.7 =
+73% of K energy) → K direction info lands below the 2-bit centroid gap.
+Outlier channels are the lowest-freq RoPE dims → error is common-mode at 4K
+(looks fine), phase-spreads by 8-32K → blow-up. SmolLM3 (no QK-norm) and
+Gemma4 (constant γ, mostly SWA) pass the same gate.
+
+**Empirical matrix (CPU, ctx 8K, chunk-1 PPL, ref q8/q8 = 18.03):**
+
+| ctk | ctv | PPL | verdict |
+|---|---|---|---|
+| turbo2 | turbo2 | 442.2 | broken |
+| turbo2 | q8_0 | 404.2 | broken → **K is the culprit** |
+| q8_0 | turbo2 | 18.12 (Δ+0.09) | **clean → V tolerates 2-bit** |
+| q4_0 | turbo2 | 19.03 (Δ+1.00) | fails 0.5 gate — K needs ≥8-bit |
+
+No zero-code ctx win for qwen: the only clean mix (q8K+turbo2V, 49.5 KB/tok)
+is bigger than q4/q4 (40.5). Also: turbo2-K costs 2.6× CPU decode; turbo2-V free.
+
+**InnerQ (fork's per-channel K equalizer): broken as shipped.** Widened its
+[0.5,2.0] clamp to 64× (commit on fix/innerq-clamp) — but GPU PPL gate shows
+the compensation path is quantitatively wrong: strength 0.001 → 431 (= no-op
+control), 0.5 → 2482 (author defaults, WORSE than off), 1.0 → 15M. Error grows
+superlinearly with scale strength; not the calibration-window mismatch (chunk 2
+equally broken). The 2× clamp was hiding a real compensation bug — feature is
+off-by-default for a reason. Real fix = load-time static per-LAYER scales
+derived from attn_k_norm γ (InnerQ state is global-128ch, γ outliers are
+per-layer) + verified Q/V compensation — parked, see fix ladder in the
+agents' reports (session scratchpad) if resumed.
 
 Gotchas hit:
 - `--n-gpu-layers 0` is NOT CPU-only on a CUDA build: it still cudaMallocs a

swap-stack/config.yaml

@@ -60,14 +60,15 @@ models:
 
   "ornith-35b-duo":
     name: "Ornith 35B (duo main)"
-    description: "Duo main: fully CPU, 64K ctx (GDN hybrid — KV only 680MB q8 in RAM). CUDA hidden — even ngl 0 tries a ~1GB pp compute buffer"
+    description: "Duo main: fully CPU, 128K ctx, q8 K + turbo2 V (K-side turbo costs 3x CPU decode, V-side is free — probe-verified; full-turbo2 quality gate Δ0.196 upper-bounds this mix). ~880MB KV. CUDA hidden — even ngl 0 tries a ~1GB pp compute buffer"
     env:
       - "CUDA_VISIBLE_DEVICES="
     cmd: |
-      ${server-base} ${q8-kv}
+      ${server-base}
+      --cache-type-k q8_0 --cache-type-v turbo2
       --model /models/deepreinforce-ai_Ornith-1.0-35B-Q2_K_L.gguf
       --n-gpu-layers 0 --jinja
-      --ctx-size 65536
+      --ctx-size 131072
       --batch-size 1024 --ubatch-size 512
       --cont-batching --parallel 1
     ttl: 0