Add a slot-cache-proxy service (profile stack) in front of llama-swap-stack.
Injects return_progress into streaming chat, taps llama.cpp prompt_progress
SSE, serves GET /progress {loading|prefill|generating|idle} with pct/tok_s/
ETA. The pi-load-progress extension polls it; point pi llmruntime baseUrl at
http://localhost:48090/v1. host net + pid:host (load tracker reads the
containerised llama-server /proc/io). Transparent superset of llama-swap
(no endpoint lost; /progress is net-new). Script mirrors llmruntime-x570.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a 4th binary to the swap-stack: PrismML llama-server-ternary (Q1_0_g128
ternary kernels, built sm75/CUDA-12.8 static) for Bonsai-27B-Q1_0.
Config: 3.8GB weights ALL on the 4GB GPU (ngl 99) + KV in CPU RAM
(--no-kv-offload, since VRAM cannot hold both) + q4_0 KV + --flash-attn
(GPU attention; iq4_nl forces CPU attention) + tiny batch (-b 64) to fit
the ~100MB VRAM headroom. Fits at 3611/3718 MiB. ~7 t/s decode, ~27 pp
prefill (1650 Ti has no tensor cores + KV over PCIe = the ceiling).
Measured alt configs (all worse): ngl48+KV-VRAM = 8.5 t/s decode but only
10 pp prefill (CPU layers tank prefill); ngl99+KV-VRAM OOMs even at ctx 2048.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Periodic slot saves queue on the serving slot and blocked in-flight
requests (user hit connection timeouts). Prefix reuse via --cache-reuse
+ ttl 0 does not need the sidecar and stays; duo-warmboot.sh remains
for manual restore after stack restarts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stack restart after the sidecar's boot pass left duo_up=0 and the heal
branch (which required exactly one member) never fired. Heal now also
triggers when zero duo members run AND the running list is empty, so a
bounced stack repopulates automatically. A deliberate swap to a non-duo
model still isn't fought (running list non-empty).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Suspend/resume kills the GPU-side duo member (CUDA context lost; often
wedged nvidia_uvm on this machine). Daemon now ticks every 60s: if
exactly one duo member is running, the missing one is reloaded via its
/upstream health endpoint and its snapshot restored. Saves stay at
5-min cadence. Boot restore skips members that are already running.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
curlimages/curl sidecar (profile stack, depends_on healthy): restores
duo slot caches on boot (pre-loads both duo servers) and auto-saves
every 300s while loaded. No manual duo-warmboot.sh calls needed; the
script remains for ad-hoc use. Snapshot tracks the latest real session;
prefix matching (--cache-reuse) handles divergent new sessions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
--slot-save-path + --cache-reuse 256 on both duo members, ./kv-cache
mounted at /cache. Measured on ornith (3K-token system prompt): warm
prefix reuse 296ms vs 106.7s cold prefill (361x); slot save 84MB in
59ms, GDN recurrent state serializes. scripts/duo-warmboot.sh wraps
save/restore/status via llama-swap /upstream passthrough. Restore
after restart: run 'duo-warmboot.sh restore' post stack boot.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AnythingLLM (port 3001, profile webui) points at llama-swap-stack:8080
via generic-openai, default model ornith-35b-duo (131K ctx). Deleted the
two >RAM IQ4 GGUFs (36.5GB — thrash-only until 64GB RAM upgrade) and
their swap-stack entries; re-download paths noted in config comment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FINDINGS answered their questions: ik_llama -23% decode / +13% prefill,
upstream master parity with turboquant on K-quant decode. Binaries stay
in the image, macros kept for future experiments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Probe matrix proved K is the broken side of turbo KV on Qwen3-4B
(QK-norm gamma outliers vs PolarQuant's no-per-channel-range format)
while V tolerates 2-bit for free. Applied to ornith duo main: K q8_0 +
V turbo2 = 128K ctx in ~880MB RAM at full speed (8.7-10.1 t/s concurrent,
vs 3x decode loss with turbo2 K). Qwen duo stays q4/q4 24K — only clean
mix is bigger than q4/q4. InnerQ equalizer confirmed broken as shipped
(compensation error grows with scale strength); documented in
MOE-FINDINGS with the full matrix and fix ladder.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Duo group (resident main+subagent for pi): flip VRAM to the small model.
Ornith experts never touch VRAM; its dense-on-GPU split starved qwen to
176 MiB / 5.8 t/s concurrent. After flip: qwen 43 t/s, ornith 8 t/s.
CUDA_VISIBLE_DEVICES= required for ornith — ngl 0 still allocates ~1GB
pp compute buffer on CUDA builds (OOM+segfault). Duo section in
MOE-FINDINGS.md; also snapshots prior swap-stack migration state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Compose shlex-splits 'command: |' block scalar into a list when used with
'entrypoint: ["/bin/sh","-c"]'. Docker then runs '/bin/sh -c exec' where
'exec' is the only -c argument and '/app/llama-server' becomes $0. 'exec'
with no program in sh exits 0 immediately → 37-restart crash-loop, no server.
Fix: use 'entrypoint: []' and 'command: [/bin/sh, -c, <|block>]' so the full
shell command is passed as a single list element — not further split by Compose.
mozempkandClaude Opus 4.8