swap-stack: slot KV save/restore for duo prompt caches (x570 S5 pattern)
--slot-save-path + --cache-reuse 256 on both duo members, ./kv-cache mounted at /cache. Measured on ornith (3K-token system prompt): warm prefix reuse 296ms vs 106.7s cold prefill (361x); slot save 84MB in 59ms, GDN recurrent state serializes. scripts/duo-warmboot.sh wraps save/restore/status via llama-swap /upstream passthrough. Restore after restart: run 'duo-warmboot.sh restore' post stack boot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -71,6 +71,7 @@ models:
|
||||
--ctx-size 131072
|
||||
--batch-size 1024 --ubatch-size 512
|
||||
--cont-batching --parallel 1
|
||||
--slot-save-path /cache/ornith-duo --cache-reuse 256
|
||||
ttl: 0
|
||||
|
||||
"qwen3-4b-duo":
|
||||
@@ -86,6 +87,7 @@ models:
|
||||
--ctx-size 24576
|
||||
--batch-size 512 --ubatch-size 256
|
||||
--cont-batching --parallel 1
|
||||
--slot-save-path /cache/qwen-duo --cache-reuse 256
|
||||
ttl: 0
|
||||
|
||||
# ── MoE offload models (2026-07-10) ────────────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user