Add a slot-cache-proxy service (profile stack) in front of llama-swap-stack.
Injects return_progress into streaming chat, taps llama.cpp prompt_progress
SSE, serves GET /progress {loading|prefill|generating|idle} with pct/tok_s/
ETA. The pi-load-progress extension polls it; point pi llmruntime baseUrl at
http://localhost:48090/v1. host net + pid:host (load tracker reads the
containerised llama-server /proc/io). Transparent superset of llama-swap
(no endpoint lost; /progress is net-new). Script mirrors llmruntime-x570.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a 4th binary to the swap-stack: PrismML llama-server-ternary (Q1_0_g128
ternary kernels, built sm75/CUDA-12.8 static) for Bonsai-27B-Q1_0.
Config: 3.8GB weights ALL on the 4GB GPU (ngl 99) + KV in CPU RAM
(--no-kv-offload, since VRAM cannot hold both) + q4_0 KV + --flash-attn
(GPU attention; iq4_nl forces CPU attention) + tiny batch (-b 64) to fit
the ~100MB VRAM headroom. Fits at 3611/3718 MiB. ~7 t/s decode, ~27 pp
prefill (1650 Ti has no tensor cores + KV over PCIe = the ceiling).
Measured alt configs (all worse): ngl48+KV-VRAM = 8.5 t/s decode but only
10 pp prefill (CPU layers tank prefill); ngl99+KV-VRAM OOMs even at ctx 2048.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
--slot-save-path + --cache-reuse 256 on both duo members, ./kv-cache
mounted at /cache. Measured on ornith (3K-token system prompt): warm
prefix reuse 296ms vs 106.7s cold prefill (361x); slot save 84MB in
59ms, GDN recurrent state serializes. scripts/duo-warmboot.sh wraps
save/restore/status via llama-swap /upstream passthrough. Restore
after restart: run 'duo-warmboot.sh restore' post stack boot.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AnythingLLM (port 3001, profile webui) points at llama-swap-stack:8080
via generic-openai, default model ornith-35b-duo (131K ctx). Deleted the
two >RAM IQ4 GGUFs (36.5GB — thrash-only until 64GB RAM upgrade) and
their swap-stack entries; re-download paths noted in config comment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FINDINGS answered their questions: ik_llama -23% decode / +13% prefill,
upstream master parity with turboquant on K-quant decode. Binaries stay
in the image, macros kept for future experiments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Probe matrix proved K is the broken side of turbo KV on Qwen3-4B
(QK-norm gamma outliers vs PolarQuant's no-per-channel-range format)
while V tolerates 2-bit for free. Applied to ornith duo main: K q8_0 +
V turbo2 = 128K ctx in ~880MB RAM at full speed (8.7-10.1 t/s concurrent,
vs 3x decode loss with turbo2 K). Qwen duo stays q4/q4 24K — only clean
mix is bigger than q4/q4. InnerQ equalizer confirmed broken as shipped
(compensation error grows with scale strength); documented in
MOE-FINDINGS with the full matrix and fix ladder.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Duo group (resident main+subagent for pi): flip VRAM to the small model.
Ornith experts never touch VRAM; its dense-on-GPU split starved qwen to
176 MiB / 5.8 t/s concurrent. After flip: qwen 43 t/s, ornith 8 t/s.
CUDA_VISIBLE_DEVICES= required for ornith — ngl 0 still allocates ~1GB
pp compute buffer on CUDA builds (OOM+segfault). Duo section in
MOE-FINDINGS.md; also snapshots prior swap-stack migration state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mozempkandClaude Opus 4.8