# Benchmarking Findings

Hardware: GTX 1650 Ti Mobile (Turing/SM75, 3717 MiB VRAM, CC 7.5) + i7-10750H 6c/12t, 15 GiB DDR4-2933 RAM.  
All benchmarks: llama.cpp `local/llama-cpp-turboquant:*-cuda-sm75-mmq` image (TurboQuant fork, `DGGML_CUDA_FORCE_MMQ=ON`).  
Date: 2026-05-05 / 2026-05-06.

---

## 1. FORCE_MMQ — Free +6–11% on Turing GPUs

**Finding**: GPUs without tensor cores (Turing = RTX 1650, 1660, 2060 etc.) run the GEMM path through cuBLAS GEMM, which is slower than the hand-written MMQ (matrix-multiply quantized) kernel. Compiling with `DGGML_CUDA_FORCE_MMQ=ON` forces the MMQ path unconditionally.

| Model | Standard image t/s | TurboQuant t/s | Gain |
|---|---|---|---|
| SmolLM3-3B | ~49.9 | 53.1 | +6.2% |
| Gemma4-E2B | ~55.7 | 61.7 | +10.7% |
| Gemma4-E4B | ~27.0 | 30.0 | +11.4% |
| Qwen3-4B | ~36.7 | 38.8 | +5.7% |

**Caution**: On Ampere/Ada/Hopper (RTX 3000+/4000+), tensor cores are faster. `FORCE_MMQ` would *hurt* on those cards. This image is SM75-only.

---

## 2. KV Quantization — turbo2 is the best sweet spot

**Finding**: The TurboQuant fork adds 2/3/4-bit KV quantization ("turbo2/3/4") beyond llama.cpp's built-in q8_0/q4_0. turbo2 at 2 bits is roughly half the size of q4_0, with acceptable perplexity loss.

**Perplexity delta vs f16 baseline** (quality gate: Δ < 0.5):

| KV type | SmolLM3-3B | Gemma4-E2B | Gemma4-E4B | Qwen3-4B |
|---|---|---|---|---|
| q8_0 | ✓ | ✓ | ✓ | ✓ |
| q4_0 | ✓ | ✓ | ✓ | ✓ |
| turbo2 | ✓ | ✓ | ✓ | **✗ BROKEN** |
| turbo3 | ✓ | ✓ | ✓ | **✗ BROKEN** |
| turbo4 | ✓ | ✓ | ✓ | **✗ BROKEN** |

### ⚠️ Critical: turbo2/3/4 breaks Qwen3-4B

Qwen3-4B uses full GQA (32 KV heads, 40 KB/token). At ctx ≥ 8192, turbo KV quantization causes catastrophic PPL degradation:

```
ctx=4096   turbo2: PPL=1.79  (baseline 1.76, Δ=0.03 ✓)
ctx=8192   turbo2: PPL=4.2   (Δ=2.4 ✗)
ctx=16384  turbo2: PPL=15.4  (Δ=13.7 ✗)
ctx=32768  turbo2: PPL=438   (broken)
```

**Never use turbo2/3/4 for Qwen3-4B.** Use q4_0.

---

## 3. MQA Architecture — Gemma4 E2B/E4B KV is tiny

**Finding**: Gemma4's hybrid attention uses Multi-Query Attention (MQA) for most layers — only 1 KV head is maintained per token instead of full GQA. This results in dramatically smaller KV cache:

| Model | KV bytes/token (q4_0) | Architecture |
|---|---|---|
| SmolLM3-3B | ~19.8 KB | GQA |
| Qwen3-4B | ~39.6 KB | full GQA |
| Gemma4-E4B | ~4.5 KB | MQA-like (42 layers) |
| Gemma4-E2B | ~1.7 KB | MQA (35 layers) |

**Implication**: E2B can hold 393K tokens in KV cache with only 651 MiB RAM. E4B can hold 163K tokens with 346 MiB RAM.

### ⚠️ turbo2 is *worse* for E2B (MQA padding artifact)

turbo2 uses block quantization. For MQA models with tiny KV tensors, the per-block header/padding overhead is proportionally larger than the savings. At E2B:

```
ctx=32768  q4_0: 57 MiB KV   turbo2: 68 MiB KV  (+19% worse!)
```

**Do not use turbo2 for Gemma4-E2B bigctx.** Use q4_0.

---

## 4. -nkvo (KV in RAM) — Massive Context Gain at PCIe Cost

**Finding**: `--no-kv-offload` moves the KV cache from VRAM to host RAM. VRAM is then entirely free for model weights and compute. The tradeoff is token generation speed — each token generation requires reading the full KV cache over PCIe x4.

**Bandwidth model**: `t/s = 1000 / (gpu_ms_empty + ctx × kv_bytes_per_tok / pcie_bw_bps × 1000)`

PCIe x4 Gen3 ≈ **8 GB/s** practical (measured from BW model fit to actual results).

### Context gains with -nkvo (v4, TurboQuant):

| Model | Pure-GPU ctx | -nkvo q4_0 rec | -nkvo turbo2 rec | KV type used |
|---|---|---|---|---|
| SmolLM3-3B | 24576 | 32768 | **65536** | turbo2 |
| Gemma4-E2B | 24576 | **393216** | 393216 | q4_0 (turbo2 worse!) |
| Gemma4-E4B | 24576 | 98304 | **163840** | turbo2 |
| Qwen3-4B | 16384 | **24576** | BROKEN | q4_0 |

Recommendation threshold: ≥ 15 t/s at 50% context fill.

### ⚠️ Qwen3.5-9B cannot use -nkvo

Qwen3.5-9B (Q8_0, 8.86 GB) with ngl=11 fills nearly all 15 GiB RAM with model weights + system overhead. At any tested context size, `-nkvo` OOMs. The existing server config at ctx=32768 with turbo2 KV in VRAM is the only viable option.

---

## 5. Qwen3.5-9B — RAM-bound, llama-perplexity incompatible

**Finding**: This model has a hybrid architecture: 8 full-attention layers + 24 linear-attention layers. The linear-attention layers cause `llama-perplexity` to fail (not OOM — the evaluation tool simply can't handle the architecture). The server works correctly.

**Performance ceiling**: Theoretical max t/s = RAM_BW / model_size = 45 GB/s ÷ 8.86 GB = **5.1 t/s**. Achieved: 4.38 t/s = 86% efficiency. This is purely RAM-bandwidth-limited.

**Thread optimization** (i7-10750H, 6 physical / 12 logical):
- Optimal: `THREADS=6` (one per physical core)
- HT hurts: t=8 → 4.22 t/s (worse than t=6 → 4.38 t/s)

---

## 6. Gemma4-E4B — all layers fit when VRAM is free

**Surprise**: E4B's Q4_K_M file is 4.7 GB — larger than the 3.7 GB VRAM. However, model weight loading is paged; at ngl=42, ALL 42 layers fit in VRAM during inference because llama.cpp holds only the needed tensors. The "file size > VRAM" heuristic is wrong for split configs.

ngl sweep result:
```
ngl=28  →  59 pp / 16.5 tg t/s
ngl=35  →  101 pp / 24.6 tg t/s
ngl=42  →  133 pp / 32.0 tg t/s  ← all layers, much faster
```

**Caution**: ngl=42 fails if another container is holding VRAM. Always stop other services before starting E4B.

---

## 7. Flash Attention (+2–3% pp, required for bigctx)

`--flash-attn on` is required for `-nkvo` bigctx profiles (prefill OOM otherwise at large ctx). For standard pure-GPU profiles it gives a small speed boost (~2–3% pp, ~1% tg). Always enable it.

---

## 8. Benchmarking pitfalls

### False OOM from prefill timeout
Early test scripts ran `llama-perplexity` on a full wiki dataset. At large contexts, prefill takes >600s and the script misread the timeout as OOM. Fix: use a 64-line "tiny" file for alloc checks — the model allocates the full KV cache at startup, then exits after trivial compute (< 15s).

### kv/tok measurement anomalies
The `kv_bytes_per_tok` column in cpu_ctx_test.sh is computed as `kv_mib / ctx`. At small ctx, block padding dominates and the value appears higher. The true per-token cost stabilizes at larger ctx. Use ctx ≥ 32768 values for BW model calibration.

---

## Summary: Recommended configurations

| Model | Profile | KV type | CTX | t/s@base | Notes |
|---|---|---|---|---|---|
| SmolLM3-3B | pure-GPU | q8_0 | 24576 | ~53 | max VRAM ctx |
| SmolLM3-3B | bigctx | turbo2 | 65536 | ~15@50% | 714 MiB RAM |
| Gemma4-E2B | pure-GPU | f16 | 24576 | ~62 | MQA = tiny KV |
| Gemma4-E2B | bigctx | q4_0 | 393216 | ~17@50% | 651 MiB RAM, turbo2 worse |
| Gemma4-E4B | pure-GPU | q4_0 | 24576 | ~30 | ngl=42 all layers |
| Gemma4-E4B | bigctx | turbo2 | 163840 | ~18@50% | 346 MiB RAM |
| Qwen3-4B | pure-GPU | q4_0 | 16384 | ~39 | NO turbo KV ever |
| Qwen3-4B | bigctx | q4_0 | 24576 | ~11@50% | turbo2 broken |
| Qwen3.5-9B | pure-GPU | turbo2 | 32768 | ~4.4 | RAM-bound, no bigctx |