mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-10-29 07:07:59 +01:00
The MoE models have a mul_mat_vec with very small m (32, 64, 128) right before the topk_moe selection. Running multiple rows per wg doesn't utilize the SMs well. I think even for larger m, f32 is so bandwidth-limited that running multiple rows doesn't help.