# Architecture

Hardware: GTX 1650 Ti Mobile (SM75/Turing, 3717 MiB VRAM) + i7-10750H 6c/12t + 15 GiB DDR4-2933 RAM.

---

## Docker Compose Architecture

### Image Strategy

Two custom images built from the [TurboQuant fork](https://github.com/TheTom/llama-cpp-turboquant) of llama.cpp:

| Image | Target | Used by |
|---|---|---|
| `local/llama-cpp-turboquant:server-cuda-sm75-mmq` | `server` | All llama-server services |
| `local/llama-cpp-turboquant:full-cuda-sm75-mmq` | `full` | All bench/test services |

Both built with `CUDA_DOCKER_ARCH: "75 -DGGML_CUDA_FORCE_MMQ=ON"`:
- SM75 = Turing architecture codepath (no tensor cores)
- `FORCE_MMQ` = always use hand-written MMQ kernels instead of cuBLAS GEMM
- `full` target includes `llama-bench`, `llama-perplexity`, `llama-cli` alongside the server

Both images share the same custom entrypoint wrapper that enables the `turbo2/3/4` KV quantization types unavailable in upstream llama.cpp. **All `docker run` calls must use `--entrypoint=""` to bypass the wrapper.**

### Compose Structure

```
compose.yaml
├── x-gpu           — NVIDIA runtime + capability passthrough (merged into all services)
├── x-hc            — Common healthcheck (curl /health, start_period overridden per service)
├── x-server        — Merged into all server services:
│   ├── volumes: ./models:/models:ro
│   ├── ports: 8080:8080
│   ├── network alias: llama-current (all servers share this alias)
│   ├── entrypoint: llama-server with $$VAR shell expansion from env_file
│   └── restart: unless-stopped
└── x-bench         — Merged into all bench services:
    ├── volumes: ./models:/models:ro, ./benchmark-results:/results, ./scripts:/scripts:ro
    └── entrypoint: /bin/bash /scripts/benchmark.sh (overrideable)
```

### Profile System

Docker Compose profiles allow mutually exclusive model selection. Only one model server should run at a time (single GPU).

```
docker compose --profile <PROFILE> up -d
```

**Server profiles** (bring up `llama-server` on port 8080):

| Profile | Model | Image | VRAM | Strategy |
|---|---|---|---|---|
| `qwen35-9b` | Qwen3.5-9B Q8_0 | TurboQuant (built) | 3.4 GB (11 layers) | RAM-bound; mlock pins weights |
| `gemma4-e2b` | Gemma4-E2B Q4_K_M | TurboQuant | ~3.4 GB | Full GPU, MQA |
| `gemma4-e4b` | Gemma4-E4B Q4_K_M | TurboQuant | ~3.5 GB | Full GPU (42 layers, CPU-split) |
| `smollm3-3b` | SmolLM3-3B Q4_K_M | TurboQuant | ~2.0 GB | Full GPU |
| `qwen3-4b` | Qwen3-4B Q4_K_M | TurboQuant | ~2.5 GB | Full GPU |

**Bigctx profiles** (server with `-nkvo`: KV cache in host RAM):

| Profile | Model | KV type | CTX | ~t/s@50% ctx |
|---|---|---|---|---|
| `smollm3-3b-bigctx` | SmolLM3-3B | turbo2 | 65536 | 15.2 |
| `gemma4-e2b-bigctx` | Gemma4-E2B | q4_0 | 393216 | 17.0 |
| `gemma4-e4b-bigctx` | Gemma4-E4B | turbo2 | 163840 | 17.8 |
| `qwen3-4b-bigctx` | Qwen3-4B | q4_0 | 24576 | 11.2 |

**Bench profiles** (one-shot benchmark containers):

| Profile | Service | Purpose |
|---|---|---|
| `bench-qwen35-9b` | bench-qwen35-9b | Also hosts `cpu_ctx_test.sh` / `kv_quant_test.sh` (all models have model files accessible) |
| `bench-gemma4-e2b` | bench-gemma4-e2b | E2B bench |
| `bench-gemma4-e4b` | bench-gemma4-e4b | E4B bench |
| `bench-smollm3-3b` | bench-smollm3-3b | SmolLM3 bench |
| `bench-qwen3-4b` | bench-qwen3-4b | Qwen3-4B bench |

**Add-on profile** (combine with any model):

| Profile | Service | Purpose |
|---|---|---|
| `webui` | openwebui | Open WebUI connecting to `llama-current:8080` |

### Env File Architecture

Each model has a dedicated `envs/.env.<model>` file injected into the container. Shell variables use `$$VAR` in the compose command to escape compose interpolation — the container shell expands them at runtime.

```
envs/
├── .env.smollm3-3b         ← pure-GPU: q8_0 KV, ctx=24576
├── .env.smollm3-3b-bigctx  ← -nkvo:    turbo2 KV, ctx=65536
├── .env.gemma4-e2b         ← pure-GPU: f16 KV, ctx=24576
├── .env.gemma4-e2b-bigctx  ← -nkvo:    q4_0 KV, ctx=393216 (turbo2 worse for MQA)
├── .env.gemma4-e4b         ← pure-GPU: q4_0 KV, ctx=24576, ngl=42
├── .env.gemma4-e4b-bigctx  ← -nkvo:    turbo2 KV, ctx=163840, ngl=42
├── .env.qwen3-4b           ← pure-GPU: q4_0 KV, ctx=16384 (NO turbo2 ever)
├── .env.qwen3-4b-bigctx    ← -nkvo:    q4_0 KV, ctx=24576 (NO turbo2 ever)
└── .env.qwen35-9b          ← mixed: turbo2 KV, ctx=32768, ngl=11, mlock
```

Key env variables per file:

```bash
MODEL_FILE          # filename under /models/
N_GPU_LAYERS        # ngl: how many transformer layers offloaded to GPU
CTX_SIZE            # context window size
THREADS / THREADS_BATCH
BATCH_SIZE / UBATCH_SIZE
CACHE_TYPE_K/V      # KV quantization: f16 | q8_0 | q4_0 | turbo2
PARALLEL            # number of concurrent request slots
EXTRA_ARGS          # passed verbatim to llama-server (e.g. --flash-attn on --no-kv-offload)
```

---

## Test Script Architecture

All test scripts run inside the `bench-qwen35-9b` container (has `full` image with all binaries), with all model files accessible via `/models/`.

### scripts/kv_quant_test.sh

**Purpose**: Determine optimal KV quantization type for each model at various context sizes.  
**Method**: `llama-perplexity` on a 4000-line synthetic text file. Computes perplexity for each (model, ctx, KV type) combination, measures Δ vs f16 baseline.  
**Quality gate**: Δ < 0.5 → acceptable; Δ ≥ 0.5 → degraded.

```
for each model:
  for each ctx in CTX_CANDIDATES:
    run f16 baseline → get PPL_baseline
    for each KV type in MODEL_KV_TYPES:
      run with that KV type → get PPL
      report Δ = PPL - PPL_baseline
```

**Outputs**:
- Pass/fail per (model, ctx, KV type) combination
- Recommendation: highest-quality KV type that stays within quality gate at all tested ctx

**Known limitations**:
- `Qwen3.5-9B`: hybrid linear-attention architecture is incompatible with `llama-perplexity` → always fails. Not a real model issue; the server works correctly.
- At very small ctx (< 4096), block-padding overhead inflates turbo2 apparent per-token cost.

### scripts/cpu_ctx_test.sh

**Purpose**: Find maximum viable context size when using `-nkvo` (KV in host RAM), accounting for PCIe bandwidth penalty.  
**Method**: Two-phase per (model, ctx, KV type):

1. **Alloc check** (fast, ~15s): run `llama-perplexity` on a 64-line file with `-nkvo`. The model allocates full KV at startup regardless of input length. If it exits cleanly → alloc succeeds; timeout/error → OOM.

2. **Speed estimation** (analytic bandwidth model):
   ```
   GPU-compute models (smollm3, e2b, e4b, qwen3-4b):
     t/s(ctx) = 1000 / (1000/baseline + ctx × kv_bytes_per_tok / PCIe_BW × 1000)
     PCIe_BW = 8 GB/s  (PCIe x4 Gen3 practical)

   RAM-bound models (qwen35-9b, ngl=11):
     t/s(ctx) = 1000 / (1000/baseline + ctx × kv_bytes_per_tok / RAM_BW × 1000)
     RAM_BW = 45 GB/s  (DDR4-2933)
   ```

3. **Recommendation**: highest ctx where `t/s@50%fill ≥ 15`.

**kv_bytes_per_tok** measured empirically: `KV_MiB_allocated / ctx_size` from actual alloc run.

**KV types tested per model**:

| Model | KV types | Reason |
|---|---|---|
| smollm3, e2b, e4b | q4_0 + turbo2 | Both safe (PPL gate passes) |
| qwen3-4b | q4_0 only | turbo2 breaks at ctx≥8192 |
| qwen35-9b | q4_0 only | OOMs regardless (skipped) |

### scripts/benchmark.sh

Default entrypoint for bench containers. Runs `llama-bench` sweep over prompt/generation lengths and thread counts, outputs CSV to `/results/`.

### scripts/quality_test.sh

Early script (superseded by kv_quant_test.sh). Tested KV types via basic generation quality comparison.

---

## Data Flow

```
Model GGUF files (./models/)
        │
        ▼
Docker container (/models/ read-only bind mount)
        │
        ├─── llama-server ──► OpenAI-compatible API on :8080
        │         │
        │    env_file values: MODEL_FILE, N_GPU_LAYERS, CTX_SIZE,
        │                     CACHE_TYPE_K/V, EXTRA_ARGS, ...
        │
        └─── llama-bench / llama-perplexity ──► benchmark-results/ (bind mount)
                  │
             test scripts (scripts/ read-only bind mount)
```

## Port / Network Layout

```
Host:8080 ──► llama_server container:8080
Host:3000 ──► open_webui container:8080 ──► http://llama-current:8080/v1 (Docker network)

llama-net (bridge):
  llama-current  — alias shared by ALL server services; only one runs at a time
```