mozempk/ggml-org-llama-cpp-mirror
1
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-11-11 14:22:09 +01:00
Code Issues Packages Projects Releases Wiki Activity
8,961 Commits 986 Branches 8,473 Tags
f42e29fdf199481effd31f281bf095ec6067757b
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Trivikram Reddy b760272f1a hexagon: guard HMX clock request for v75+ platforms (#22377) 2026-04-25 17:58:26 -07:00
Developer-Ecosystem-EngineeringandGeorgi Gerganov d1649047a3 metal : optimize Metal Tensor API usage for GGML_OP_MUL_MAT (#20962) 
* Optimize Metal Tensor API usage for matmul2d

Separates the Metal Tensor API (matmul2d) path in kernel_mul_mm into its own standalone kernel, gated by GGML_METAL_HAS_TENSOR.

The legacy simdgroup_matrix kernel is preserved under #else.

Previously both paths were interleaved via #ifdef blocks within a single kernel, forcing the tensor path to share the legacy kernel's data layout and threadgroup memory scheme. Splitting the kernel enabled memory and dispatch optimizations that weren't possible when the two paths shared code structure.

* cont : cleanup

* cont : cleanup

* cont : cleanup

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-04-25 15:14:28 +03:00
ddh0 9d34231bb8 llama-quant : default ftype param Q5_1 --> Q8_0 (#20828) 
Change the default `ftype` in `llama_model_quantize_params` from
`LLAMA_FTYPE_MOSTLY_Q5_1` to `LLAMA_FTYPE_MOSTLY_Q8_0`.

In case some external program naively uses the default quantization
params, we should probably default to a known-good type like Q8_0 rather
than Q5_1, which is rather old.
 2026-04-25 09:25:35 +03:00
Trivikram Reddy 361fe72acb Hexagon: Bump HMX Frequency to Max Corner (#22334) 
* hexagon: bump HMX freq to max corner

* hex-mm: fix error in log msg
 2026-04-24 13:55:17 -07:00
Xuan-Son Nguyen 82d3f4d3b2 mtmd: also support LLAMA_ROPE_TYPE_NONE (#22242) 2026-04-22 12:16:29 +02:00
Xuan-Son Nguyen 17f6245168 server: ignore reasoning content from transcription api (#21905) 2026-04-22 12:10:50 +02:00
Xuan-Son Nguyen 04fe84b69d server: allow cancel loading model (#21814) 2026-04-22 00:26:09 +02:00
Xuan-Son Nguyen 9998d88bc8 mtmd: correct mtmd_decode_use_mrope() (#22188) 2026-04-21 10:53:37 +02:00
Xuan-Son Nguyen 86f8daacfe mtmd: correct get_n_pos / get_decoder_pos (#22175) 2026-04-20 23:29:19 +02:00
neha-haNeha AbbasReese Levine
 		a6cc43c286 ggml-webgpu: updated matrix-vector multiplication (#21738) 
* merged properly, but slow q3_k and q5_k with u32 indexing

* Start on new mat-vec

* New format float paths working

* Working q4_0

* Work on remaining legacy q-types

* port k-quants to new matvec

* remove old shader

* Remove old constants, format

* remove accidental file

---------

Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
Co-authored-by: Reese Levine <reeselevine1@gmail.com>
 2026-04-20 07:37:17 -07:00
Xuan-Son Nguyen a678916623 mtmd: refactor mtmd_decode_use_mrope (#22161) 2026-04-20 14:45:11 +02:00
Xuan-Son Nguyen 19124078be mtmd: add pos_0 to mtmd_image_tokens_get_decoder_pos (breaking change) (#22082) 
* mtmd: add pos_0 to mtmd_image_tokens_get_decoder_pos

* fix build
 2026-04-19 11:57:21 +02:00
Xuan-Son Nguyen 089dd41fe3 cmake: use glob to collect src/models sources (#22005) 2026-04-16 23:25:16 +02:00
Xuan-Son Nguyen 4fbdabdc61 model: using single llm_build per arch (#21970) 
* model: using single llm_build per arch

* fix merge

* nits
 2026-04-16 21:10:22 +02:00
rehan-10xengineerRehan Qasimtaimur-10x
 		1e796eb41f ggml-cpu: add 128-bit RVV implementation for Quantization Vector Dot (#20633) 
* ggml-cpu: add 128-bit impls for i-quants, ternary quants

* ggml-cpu: add 128-bit impls for iq2_xs, iq3_s, iq3_xxs, tq2_0

Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai>

* ggml-cpu: refactor; add rvv checks

---------

Co-authored-by: taimur-10x <taimur.ahmad@10xengineers.ai>
Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai>
 2026-04-16 11:15:15 +03:00
rehan-10xengineerandRehan Qasim 5637536517 ggml : implemented simd_gemm kernel for riscv vector extension (#20627) 
Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai>
 2026-04-16 11:14:26 +03:00
Xuan-Son Nguyen 408225bb1a server: use random media marker (#21962) 
* server: use random media marker

* nits

* remove legacy <__image__> token

* revert special char in random
 2026-04-15 23:52:22 +02:00
Xuan-Son Nguyen 80d8770804 docs: more extensive RoPE documentation [no ci] (#21953) 
* more extensive ggml_rope documentation

* add more docs

* nits
 2026-04-15 14:45:16 +02:00
Xuan-Son NguyenandGeorgi Gerganov fae3a28070 ggml : remove ggml-ext.h (#21869) 
* ggml: correct placement of ggml-ext.h

* ggml : remove ggml-ext.h

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-04-14 17:32:58 +03:00
Xuan-Son Nguyen 707c0b7a6e mtmd: add mtmd_image_tokens_get_decoder_pos() API (#21851) 
* mtmd: add mtmd_image_tokens_get_decoder_pos() API

* consistent naming

* fix build
 2026-04-14 16:07:41 +02:00
Xuan-Son Nguyen e489a5ca0e server: support OAI /v1/audio/transcriptions API (#21863) 
* server: support OAI /v1/audio/transcriptions API

* address autoreview comments

* correct default response_format value
 2026-04-14 11:09:52 +02:00
Xuan-Son Nguyen e974923698 docs: listing qwen3-asr and qwen3-omni as supported (#21857) 
* docs: listing qwen3-asr and qwen3-omni as supported

* nits
 2026-04-13 22:28:17 +02:00
Xuan-Son Nguyen 920b3e78cb mtmd: use causal attn for gemma 4 audio (#21824) 2026-04-13 09:47:55 +02:00
Xuan-Son NguyenandSigbjørn Skjæret 21a4933042 mtmd: qwen3 audio support (qwen3-omni and qwen3-asr) (#19441) 
* add qwen3a

* wip

* vision ok

* no more deepstack for audio

* convert ASR model ok

* qwen3 asr working

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* nits

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* fix bad merge

* fix multi inheritance

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-04-12 23:57:25 +02:00
Xuan-Son Nguyen aa4695c5e5 mtmd: add gemma 4 test (vision + audio) [no ci] (#21806) 
* mtmd: add gemma 4 test (vision + audio)

* add to docs
 2026-04-12 16:29:03 +02:00
Xuan-Son Nguyen 6313acbef0 docs: add guide on how to add multimodal support (#21778) 
* docs: add guide on how to add multimodal support

* nits
 2026-04-12 13:02:38 +02:00
Xuan-Son Nguyen 057dba336e model: fix multimodal padding token for gemma3n/gemma4 (#21625) 
* model: fix multimodal padding token for gemma3n/gemma4

* nits
 2026-04-09 12:18:23 +02:00
Xuan-Son Nguyen 501aeed18f mtmd: support dots.ocr (#17575) 
* convert gguf

* clip impl

* fix conversion

* wip

* corrections

* update docs

* add gguf to test script
 2026-04-09 12:16:38 +02:00
Marxist-LeninistandMarxist-Leninist 8a65a7a8ee ci: drop v5 all: composition from labeler.yml (#21627) 
actions/labeler@v6 removed the `all:` / `any:` composition keys.
The `server/webui` and `server` entries used `all:` to combine
`any-glob-to-any-file` with negated `all-globs-to-all-files`,
which now errors on every PR with:

    Unknown config options were under "changed-files": all

Flatten both entries to a single `any-glob-to-any-file`. PRs
touching both webui and other server files will now receive both
labels instead of only `server/webui`.

Co-authored-by: Marxist-Leninist <noreply@users.noreply.github.com>
 2026-04-09 08:20:19 +02:00
Erik Scholz 3ba12fed0a kv-cache : extend cache quantization checks (#21586) 
to also check for enabled flash attention, instead of just auto.
 2026-04-08 16:08:57 +03:00
Xuan-Son Nguyen 3979f2bb08 docs: add hunyuan-ocr gguf, also add test [no ci] (#21490) 2026-04-06 14:02:37 +02:00
ddh0 5d3a4a7da5 server : fix logging of build + system info (#21460) 
This PR changes the logging that occurs at startup of llama-server.
Currently, it is redundant (including CPU information twice) and it is
missing the build + commit info.
 2026-04-05 16:14:02 +02:00
Slobodan JosicandSigbjørn Skjæret 7c7d6ce5c7 [HIP] Bump ROCm version to 7.2.1 (#21066) 
Bump ROCm version on Linux from 7.2 to 7.2.1
Add gfx1102 target
Delete LLVM workaround since ROCm 7.2.1 has fix for ROCm 7.2 perf regression https://github.com/ROCm/rocm-systems/issues/2865

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-04-03 00:59:20 +02:00
Xuan-Son Nguyen 63f8fe0ef4 model, mtmd: fix gguf conversion for audio/vision mmproj (#21309) 
* fix gguf conversion for audio/vision mmproj

* fix test
 2026-04-02 17:10:32 +02:00
Jesus TalaveraandClaude Opus 4.6 6137c325a1 chat : add Granite 4.0 chat template with correct tool_call role mapping (#20804) 
* chat : add Granite 4.0 chat template with correct tool_call role mapping

Introduce `LLM_CHAT_TEMPLATE_GRANITE_4_0` alongside the existing Granite
3.x template (renamed `LLM_CHAT_TEMPLATE_GRANITE_3_X`).

The Granite 4.0 Jinja template uses `<tool_call>` XML tags and maps the
`assistant_tool_call` role to `<|start_of_role|>assistant<|end_of_role|><|tool_call|>`.
Without a matching C++ handler, the fallback path emits the literal role
`assistant_tool_call` which the model does not recognize, breaking tool
calling when `--jinja` is not used.

Changes:
- Rename `LLM_CHAT_TEMPLATE_GRANITE` to `LLM_CHAT_TEMPLATE_GRANITE_3_X`
  (preserves existing 3.x behavior unchanged)
- Add `LLM_CHAT_TEMPLATE_GRANITE_4_0` enum, map entry, and handler
- Detection: `<|start_of_role|>` + (`<tool_call>` or `<tools>`) → 4.0,
  otherwise → 3.x
- Add production Granite 4.0 Jinja template
- Add tests for both 3.x and 4.0 template paths (C++ and Jinja)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Code review: follow standard format and use common logic in test-chat-template.cpp

* Rename custom_conversation variable for extra_conversation to give it a more meaningful name

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-02 11:28:56 +02:00
Xuan-Son Nguyen c30e012253 contrib : rewrite AGENTS.md, make it more clear about project values (#21270) 
* contrib : rewrite AGENTS.md, make it more clear about types of permitted AI usage

* permit AI for writing code
 2026-04-01 23:31:51 +02:00
Xuan-Son Nguyen 0356e33aaf scripts: add function call test script (#21234) 
* scripts: add function call test script

* add reasoning_content

* fix lint
 2026-04-01 15:31:58 +02:00
Xuan-Son NguyenandAleksander Grygier 4a00bbfed6 server: (webui) no more gzip compression (#21073) 
* webui: no more gzip

* try changing a small line

* Revert "try changing a small line"

This reverts commit 0d7a353159.

* fix lint

* fix test

* rebuild

* split into html/css/js

* lint

* chore: update webui build output

* chore: Update git hooks script

* server: update webui build output

* chore: Update pre-commit hook

* refactor: Cleanup

---------

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
 2026-03-31 15:44:26 +02:00
Xuan-Son NguyenGeorgi GerganovAleksander Grygier
 		abf9a62161 server: wrap headers for mcp proxy (#21072) 
* server: wrap headers for mcp proxy

* Update tools/server/server-cors-proxy.h

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* fix build

* chore: update webui build output

* chore: update webui build output

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
 2026-03-30 08:59:16 +02:00
Woof Dog 82b703f8bc Document custom default webui preferences in server README (#19771) 2026-03-28 14:19:16 +01:00
Ts-soundandAaron Teo bf934f28db docker : fix and enable ARM64 image build (#20929) 
* CI: fix ARM64 image build error & enable compilation

* Update .github/workflows/docker.yml

Co-authored-by: Aaron Teo <taronaeo@gmail.com>

* CI: revert ggml/src/ggml-cpu/CMakeLists.txt

* Update .github/workflows/docker.yml

Co-authored-by: Aaron Teo <taronaeo@gmail.com>

* CI: update runs-on to ubuntu24.04, and update ARM64 build image ( ubuntu_version: "24.04")

* CI: change cpu.Dockerfile gcc to 14;

* CI : cpu.Dockerfile , update pip install .

* Update .github/workflows/docker.yml

Co-authored-by: Aaron Teo <taronaeo@gmail.com>

---------

Co-authored-by: Aaron Teo <taronaeo@gmail.com>
 2026-03-28 01:45:09 +01:00
Xuan-Son Nguyen 871f1a2d2f mtmd: add more sanity checks (#21047) 2026-03-27 11:00:52 +01:00
Xuan-Son Nguyen 20197b6fe3 server: add built-in tools backend support (#20898) 
* wip: server_tools

* refactor

* displayName -> display_name

* snake_case everywhere

* rm redundant field

* change arg to --tools all

* add readme mention

* llama-gen-docs
 2026-03-27 10:07:11 +01:00
ren 9bcb4eff4d metal : Fix dimension constraint violation in matmul2d descriptor (#21048) 
Updates Metal tensor API test probe to fix the dimension constraint violation in the matmul2d descriptor (at least one value must be a multiple of 16).
 2026-03-27 09:05:21 +02:00
Xuan-Son Nguyen 8c60b8a2be ci: pin external actions to exact commit SHA (#21033) 2026-03-26 20:44:00 +01:00
Xuan-Son Nguyen a73bbd5d92 mtmd: refactor image preprocessing (#21031) 
* mtmd: refactor image pre-processing

* correct some places

* correct lfm2

* fix deepseek-ocr on server

* add comment to clarify about mtmd_image_preprocessor_dyn_size
 2026-03-26 19:49:20 +01:00
Xuan-Son NguyenandSigbjørn Skjæret 914eb5ff0c jinja: fix macro with kwargs (#20960) 
* jinja: fix macro with kwargs

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* fix newline problem

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-03-25 12:22:48 +01:00
Xuan-Son NguyenandPiotr Wilkin bd6992180b contrib: add "Requirements" section to PR template (#20841) 
* contrib: add "Requirements" section to PR template

* typo [no ci]

* use h2, add "Additional information"

---------

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
 2026-03-23 16:59:02 +01:00
Xuan-Son Nguyen 31a5cf4c3f server: use httplib dynamic threads (#20817) 
* server: use httplib dynamic threads

* change to n_threads_http + 1024
 2026-03-23 12:22:46 +01:00
Jhen-Jie Hong 7a0b6a635e common/autoparser : detect reasoning markers when enable_thinking changes system prompt (#20859) 2026-03-23 08:35:27 +01:00
Xuan-Son Nguyen 49bfddeca1 server: allow router to report child instances sleep status (#20849) 
* server: allow router to report child instances sleep status

* refactor

* move sleeping to state

* nits
 2026-03-22 18:33:52 +01:00
Xuan-Son Nguyen 568aec82d2 docs : explicit about banning accounts that violates policy (#19593) 2026-03-21 15:50:16 +01:00
shalinib-ibm e6ec21e62f ggml-cpu: add always_inline to tinyBLAS_PPC accumulator saves (#20791) 
Explicitly mark save_acc and add_save_Acc with always_inline
in tinyBLAS_PPC. This ensures the compiler keeps MMA accumulator
disassembly within kernel's register context, preventing un-necessary
stask spills.

Signed-off-by: Shalini Salomi Bodapati <Shalini.Salomi.Bodapati@ibm.com>
 2026-03-21 07:11:45 +08:00
Xuan-Son NguyenandGeorgi Gerganov fb78ad29bb server: (doc) clarify in-scope and out-scope features (#20794) 
* server: (doc) clarify in-scope and out-scope features

* Apply suggestions from code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-03-20 14:03:50 +01:00
Xuan-Son Nguyen 1e64534570 mtmd: add clip_graph::build_mm() (#20751) 
* clip: add build_mm()

* apply to all models

* add TODO for bias overload
 2026-03-19 13:11:39 +01:00
Krishna SridharandMax Krasnyansky cf23ee2447 hexagon: add neg, exp, sigmoid, softplus ops, cont, repeat ops (#20701) 
Add element-wise unary ops needed by Qwen 3.5's DeltaNet linear
attention layers. These ops follow the existing unary-ops pattern
with VTCM DMA double-buffering.

- neg: negate via scale by -1.0
- exp: uses existing hvx_exp_f32 HVX intrinsics
- sigmoid: uses existing hvx_sigmoid_f32_aa HVX intrinsics
- softplus: log(1 + exp(x)) scalar fallback
- CONT reuses the existing CPY infrastructure since making a tensor
  contiguous is equivalent to a same-type copy.
- REPEAT implements tiled memory copy with multi-threaded execution via
  the worker pool, supporting f32 and f16 types. The kernel parallelizes
  across output rows and uses memcpy for each tile.

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
 2026-03-17 15:34:36 -07:00
Xuan-Son NguyenandSigbjørn Skjæret d34ff7eb5b model: mistral small 4 support (#20649) 
* model: mistral small 4 support

* fix test

* fix test (2)

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update convert_hf_to_gguf.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* change newline

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-03-17 00:31:14 +01:00
Woof Dog d8c331c0af webui: use date in more human readable exported filename (#19939) 
* webui: use date in exported filename

Move conversation naming and export to utils

update index.html.gz

* webui: move literals to message export constants file

* webui: move export naming and download back to the conversation store

* chore: update webui build output

* webui: add comments to some constants

* chore: update webui build output
 2026-03-16 11:18:13 +01:00
Xuan-Son Nguyen 94d0262277 mtmd: add llama-mtmd-debug binary (#20508) 
* mtmd: add llama-mtmd-debug binary

* adapt

* fixes

* fix compile error

* fix windows compile error

* rm legacy clip_debug_encode()

* add MTMD_API to fix build
 2026-03-14 15:52:29 +01:00
rehan-10xengineerRehan Qasimtaimur-10xRehan Qasim
 		fbaa95bc29 ggml-cpu: add RVV vec dot kernels for quantization types (#18859) 
* ggml-cpu: add rvv quantize_row_q8_K kernel

Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai>

* ggml-cpu: add rvv vec_dot for iq4_nl, mxfp4, iq2_xxs

Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai>

* ggml-cpu: add rvv vec_dot for iq4_xs, refactor

* ggml-cpu: remove ifunc for rvv vec dot

* ggml-cpu: add vec_dot for iq2_xs, iq3_xxs

Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai>

* ggml-cpu: refactor quants.c

---------

Co-authored-by: taimur-10x <taimur.ahmad@10xengineers.ai>
Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai>
Co-authored-by: Rehan Qasim <rehanbhatti0317@gmail.com>
 2026-03-13 17:36:04 +02:00
ddh0 4a748b8f15 common : fix --n-cpu-moe, --cpu-moe for models with fused gate + up (#20416) 2026-03-12 00:13:28 +01:00
ddh0 10e5b148b0 llama-quant : correct n_attention_wv usage (#20357) 
* llama-quant : correct `n_attention_wv` usage

In #19770, I introduced a regression in the way the
`quantize_state_impl` counter values were initialized. I was
incrementing and using `n_attention_wv` in the same loop, when it should
have been fixed by the time we're deciding tensor types in
`llama_tensor_get_type_impl` (for `use_more_bits`).

I never observed a difference in any of [my
tests](https://github.com/ggml-org/llama.cpp/pull/19770#issuecomment-4000424712)
- it was only after @bartowski kindly pointed this out that I realized
it was incorrect. (Thanks!)

* simplify
 2026-03-10 21:43:29 +02:00
Xuan-Son Nguyen 0842b9b465 model: fix step3.5 n_rot (#20318) 2026-03-09 23:42:24 +01:00
Xuan-Son Nguyen 59db9a357d llama: dynamic head_dim and n_rot for SWA (#20301) 
* llama: dynamic head_dim and n_rot for SWA

* also add gguf_writer wrappers

* fix build

* build_rope_shift arg reorder
 2026-03-09 22:22:39 +01:00
ddh0 b518195101 llama-quant : left-align tensor names in output (#20117) 2026-03-09 09:28:41 +02:00
shalinib-ibm c6980ff29d ggml-cpu: Fix gcc 15 ICE on ppc64le (#20083) (#20130) 
This patch addresses an Internal Compiler Error (Segmentation fault)
observed with gcc 15 by replacing the intrinsic + cast by doing
a cat on the data first and then calling the intrinsic. This bypasses the
buggy compiler path while maintaining identical instruction selection.

Performance Verification:
Assembly analysis on RHEL 9 (GCC 15.1.1) confirms that both the original
code and this fix generate the identical Power10 prefixed load instruction:
    `plxv 40, 2(14)`

This ensures zero performance regression while unblocking builds on
newer toolchains.

Reproduced on:
- Alpine Linux + GCC 15.2.0-r2
- RHEL 9  + GCC 15.1.1 (gcc-toolset-15)

Signed-off-by: Shalini Salomi Bodapati <Shalini.Salomi.Bodapati@ibm.com>
 2026-03-06 23:22:39 +08:00
ddh0 c99909dd0b impl : use 6 digits for tensor dims (#20094) 
Many models have vocabulary sizes, and thus tensor shapes, with more
than 5 digits (ex: Gemma 3's vocab size is 262,208).

I already fixed this for `llama_format_tensor_shape` but missed it for
`llama_format_tensor_shape` until now. Oops.
 2026-03-04 09:53:38 +01:00
Mickael Desgranges ecd99d6a9a docs: Fix intel documentation link (#20040) 2026-03-03 21:50:00 +08:00
Dmitry Atamanov 05728db18e vendors : update miniaudio library to 0.11.24 (#19914) 2026-02-28 16:10:01 +01:00
ddh0 832aa94762 common : add more aliases for sampler CLI params (#19797) 
* common : add more aliases for sampler CLI params
 2026-02-25 16:34:25 +01:00
Slobodan JosicandSigbjørn Skjæret 3af34b9ff5 ci : update the ROCm/HIP toolchain versions [no ci] (#19891) 
* [HIP] Update ROCm build container to rocm/dev-ubuntu-22.04:7.2 and HIP_SDK to 26.Q1

* revert container version

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-02-25 15:54:49 +01:00
Xuan-Son Nguyen c747294b2d scripts: update corpus of compare-logprobs (#19326) 
* scripts: update corpus of compare-logprobs

* fix
 2026-02-25 12:57:34 +01:00
Xuan-Son Nguyen 5452d736f8 jinja: correct stats for tojson and string filters (#19785) 2026-02-22 21:08:23 +01:00
ddh0andSigbjørn Skjæret 492bc31978 quantize : add --dry-run option (#19526) 
* clean slate for branch

* use 6 characters for tensor dims

* add --dry-run to llama-quantize

* use 6 characters for tensor dims (cont.)

* no need to re-calculate ggml_nbytes for tensor

* fix indent

* show model and quant BPW when quant completes

* add example to --help

* new function `tensor_requires_imatrix`, add courtesy warning about imatrix

* missing __func__, move imatrix flag set

* logic error

* fixup tensor_requires_imatrix

* add missing `GGML_TYPE`s

* simplify and rename `tensor_type_requires_imatrix`

* simplify for style

* add back Q2_K edge case for imatrix

* guard ftype imatrix warning

* comment ref #12557

* remove per @compilade

* remove unused `params` parameter

* move `bool dry_run` per GG

* move `bool dry_run` per GG

* Update src/llama-quant.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update src/llama-quant.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update src/llama-quant.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-02-20 09:20:16 +01:00
Ryan MangenoandSigbjørn Skjæret c0d0430340 model : full modern bert support (#18330) 
* full modern bert support

* added gelu op in rank pooling for modern bert

* still working on stuff, added mean calculation before classifier head

* Update convert_hf_to_gguf.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* first layer is dense, as per modern bert research paper

* Update src/llama-graph.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* fixed set input for mean pooling to check if pooling type is ranking since modern bert does mean & rank

* Update src/llama-graph.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update convert_hf_to_gguf.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-02-19 08:52:21 +01:00
shalinib-ibm 3bb2fcc856 llamafile: powerpc: add FP16 MMA path for Q4/Q8 matmul (#19709) 
Avoid xvi8ger4pp signed→unsigned bias correction by dequantizing Q4/Q8
inputs to FP16 and using FP16×FP16→FP32 MMA. This removes
post-processing overhead and improves performance.

Performance Impact:
1.5 ~ 2x improvement in PP_Speed for Q4 and Q8 Models,
measured with llama-bench and llama-batched-bench.
Q8 Model: granite-4.0-h-micro-Q8_0.gguf (from huggingface)
Q4 Model: Meta-Llama3-8b Q4 model (generated with llama-quantize from
f32 model)

llama-bench Q8 Model Results:
 model                          	       size 	     params 	 backend    	 threads 	            test 	Base t/s	Patch t/s
 granitehybrid 3B Q8_0          	   3.16 GiB 	     3.19 B 	 CPU        	      10 	             pp8 	         64.48 ± 4.72 	         73.99 ± 0.27
 granitehybrid 3B Q8_0          	   3.16 GiB 	     3.19 B 	 CPU        	      10 	            pp16 	         80.11 ± 0.32 	        112.53 ± 0.40
 granitehybrid 3B Q8_0          	   3.16 GiB 	     3.19 B 	 CPU        	      10 	            pp32 	         89.10 ± 0.27 	        152.95 ± 0.68
 granitehybrid 3B Q8_0          	   3.16 GiB 	     3.19 B 	 CPU        	      10 	            pp64 	         93.65 ± 0.25 	        187.83 ± 0.83
 granitehybrid 3B Q8_0          	   3.16 GiB 	     3.19 B 	 CPU        	      10 	           pp128 	         99.93 ± 0.02 	        201.32 ± 0.11
 granitehybrid 3B Q8_0          	   3.16 GiB 	     3.19 B 	 CPU        	      10 	           pp256 	        102.32 ± 0.40 	        208.32 ± 0.41
 granitehybrid 3B Q8_0          	   3.16 GiB 	     3.19 B 	 CPU        	      10 	           pp512 	        103.42 ± 0.40 	        209.98 ± 0.14
 granitehybrid 3B Q8_0          	   3.16 GiB 	     3.19 B 	 CPU        	      10 	           tg128 	         20.35 ± 0.01 	         19.57 ± 0.01

llama-bench Q4 Model Results:
 model                          	       size 	     params 	 backend    	 threads 	            test 	              Base    t/s 	               Patch   t/s
 llama 8B Q4_0                  	   4.33 GiB 	     8.03 B 	 CPU        	      10 	             pp8 	         34.77 ± 0.10 	         41.23 ± 0.08
 llama 8B Q4_0                  	   4.33 GiB 	     8.03 B 	 CPU        	      10 	            pp16 	         40.81 ± 0.04 	         64.55 ± 0.15
 llama 8B Q4_0                  	   4.33 GiB 	     8.03 B 	 CPU        	      10 	            pp32 	         44.65 ± 0.05 	         90.84 ± 0.22
 llama 8B Q4_0                  	   4.33 GiB 	     8.03 B 	 CPU        	      10 	            pp64 	         47.49 ± 0.03 	        114.39 ± 0.11
 llama 8B Q4_0                  	   4.33 GiB 	     8.03 B 	 CPU        	      10 	           pp128 	         49.29 ± 0.24 	        120.13 ± 0.19
 llama 8B Q4_0                  	   4.33 GiB 	     8.03 B 	 CPU        	      10 	           pp256 	         49.77 ± 0.23 	        121.51 ± 0.11
 llama 8B Q4_0                  	   4.33 GiB 	     8.03 B 	 CPU        	      10 	           pp512 	         49.89 ± 0.23 	        117.52 ± 0.10
 llama 8B Q4_0                  	   4.33 GiB 	     8.03 B 	 CPU        	      10 	           tg128 	         13.40 ± 0.01 	         13.37 ± 0.00

Llama perplexity Results:

Model	                    Base Final PPL Estimate	Patch Final PPL Estimate
granite-4.0-h-micro-Q8_0    1.3862 +/- 0.04424	        1.3868 +/- 0.04432
Meta-Llama3-8b Q4	    1.3801 +/- 0.04116	        1.3803 +/- 0.04116

Signed-off-by: Shalini.Salomi.Bodapati <Shalini.Salomi.Bodapati@ibm.com>
 2026-02-19 14:28:53 +08:00
Xuan-Son NguyenandSigbjørn Skjæret eeef3cfced model: support GLM-OCR (#19677) 
* model: support GLM-OCR

* Update convert_hf_to_gguf.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-02-18 17:51:40 +01:00
agent-enemy-2Jake ChavisGeorgi Gerganov
 		2d8015e8a4 llama : update LoRA API. + fix excessive graph reserves (#19280) 
* Refactoring to use new llama_put_adapter_loras

* cont : alternative lora API

---------

Co-authored-by: Jake Chavis <jakechavis6@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-02-14 10:06:27 +02:00
Xuan-Son NguyenandSigbjørn Skjæret 752584d5f5 model: support GLM MoE DSA arch (NOTE: indexer is not yet supported) (#19460) 
* model: support GLM MoE DSA arch

* working version

* pyright

* keep indexer tensors

* add indexer gguf params

* loaded now

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* update

* Update src/llama-model.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* minor fix and cleanup

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-02-13 14:56:53 +01:00
Xuan-Son Nguyen 9a96352729 test: fix IMROPE perf test case (#19465) 2026-02-10 14:37:50 +01:00
Xuan-Son Nguyen 98e57ca422 chat: fix case where template accepts type content only (#19419) 
* chat: fix case where template accepts type content only

* rm stray log

* reuse render_message_to_json
 2026-02-09 22:14:12 +01:00
will-lms af252d0758 metal : add missing includes (#19348) 2026-02-05 08:05:09 +02:00
Xuan-Son Nguyen e0c93af2a0 debug: make common_debug_print_tensor readable (#19331) 
* debug: make common_debug_print_tensor readable

* editorconfig
 2026-02-04 17:55:31 +01:00
Xuan-Son Nguyen 8abcc70a74 model: (qwen3next) correct vectorized key_gdiff calculation (#19324) 
* model: (qwen3next) correct vectorized key_gdiff calculation

* move transpose to outside of loop
 2026-02-04 13:09:58 +01:00
Xuan-Son Nguyen 07a7412a3b mtmd: add min/max pixels gguf metadata (#19273) 2026-02-02 20:59:06 +01:00
tc-mb ec6c7421e4 mtmd: support MiniCPM-o 4.5(vision only) (#19211) 
Signed-off-by: tc-mb <caitianchi@modelbest.cn>
 2026-01-30 23:19:30 +01:00
yuloandzhang hui f3dd7b8e68 HIP: add mmf for CDNA (#18896) 
* refactor mmf rows_per_block

* speed up compile

* pass cdna compile

* fix cuda error

* clean up mmf

* f32 mmf

* clean float mma

* fix mmf error

* faster mmf

* extend tile k

* fix compile error

* Revert "extend tile k"

This reverts commit 4d2ef3d483.

* fix smem overflow

* speed up compiling mmf

* speed up compile for hip

* 512 block for cdna

* config pad size

* fix as comment

* update select logic

* move some code to cuh

* fix as comment

* correct cdna3 config

---------

Co-authored-by: zhang hui <you@example.com>
 2026-01-29 11:10:53 +01:00
shalinib-ibm 7afdfc9b84 ggml-cpu: Enable FP16 MMA kernels on PPC (#19060) 2026-01-27 11:52:34 +08:00
Xuan-Son NguyenandAldehir Rojas bb02f74c61 chat: fix language input for translategemma (#19052) 
* chat: fix language input for translategemma

* Update common/chat.cpp

Co-authored-by: Aldehir Rojas <hello@alde.dev>

---------

Co-authored-by: Aldehir Rojas <hello@alde.dev>
 2026-01-24 17:58:45 +01:00
Xuan-Son Nguyen b5b8fa1c8b chat : fix translategemma crash on common_chat_format_example (#19019) 2026-01-23 12:03:42 +01:00
Xuan-Son Nguyen 51fa458a92 server : support preserving reasoning_content in assistant message (#18994) 
* support reasoning_content input

* report template caps to webui

* add docs

* rm commented code
 2026-01-22 21:30:06 +01:00
Xuan-Son Nguyen 4e595b250a server: do not log certain endpoints (avoid log spam) (#19028) 2026-01-22 19:24:37 +01:00
Xuan-Son Nguyen 9eb5bfec1a mtmd : update docs to use llama_model_n_embd_inp (#18999) 2026-01-22 14:36:32 +01:00
Xuan-Son Nguyen 2c1f199653 cli : fix reasoning responses in CLI (#18961) 
* cli : fix reasoning responses in CLI

* fix build

* fix build (2)
 2026-01-20 18:23:25 +01:00
Xuan-Son Nguyen 6df686bee6 server : refactor oai_parser_opt, move it to server_chat_params (#18937) 
* server_chat_params

* move chat format into CLI

* use meta whenever possible

* clean up, no more chatml fallback
 2026-01-19 23:28:01 +01:00
ddh0andXuan Son Nguyen 1706a6d7c6 convert : support Glm4MoeLite (#18936) 
* initial commit for branch

* add glm-4.7-flash, move tokenizer hash

* use `glm4` pretok

* silence flake8 E302 (CI)

* apply review feedback

* add <|user|> as eog

* also add EOG `<|observation|>`

* revert llama-vocab

* inherit vocab from glm4

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
 2026-01-19 23:09:20 +01:00
Lennart Austenfeld 18361c579c server: fix memory reservations in populate_token_probs (#18787) 2026-01-19 19:13:31 +01:00
Xuan-Son NguyenandSigbjørn Skjæret fe44d35574 tests : add test-jinja -py option for cross-checking (#18906) 
* tests : add test-jinja -py option or cross-checking

* Update tests/test-jinja.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* fix + add source

* SandboxedEnvironment

* fix array.map case

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-01-18 08:14:27 +01:00
Xuan-Son Nguyen f55b033ae6 jinja: correct member access rule (#18905) 2026-01-18 00:48:55 +01:00
Xuan-Son NguyenAlde RojasSigbjørn SkjæretGeorgi Gerganov
 		c15395f73c common : implement new jinja template engine (#18462) 
* jinja vm

* lexer

* add vm types

* demo

* clean up

* parser ok

* binary_expression::execute

* shadow naming

* bin ops works!

* fix map object

* add string builtins

* add more builtins

* wip

* use mk_val

* eval with is_user_input

* render gemma tmpl ok

* track input string even after transformations

* support binded functions

* keyword arguments and slicing array

* use shared_ptr for values

* add mk_stmt

* allow print source on exception

* fix negate test

* testing more templates

* mostly works

* add filter_statement

* allow func to access ctx

* add jinja-value.cpp

* impl global_from_json

* a lot of fixes

* more tests

* more fix, more tests

* more fixes

* rm workarounds

* demo: type inferrence

* add placeholder for tojson

* improve function args handling

* rm type inference

* no more std::regex

* trailing spaces

* make testing more flexible

* make output a bit cleaner

* (wip) redirect minja calls

* test: add --output

* fix crash on macro kwargs

* add minimal caps system

* add some workarounds

* rm caps_apply_workarounds

* get rid of preprocessing

* more fixes

* fix test-chat-template

* move test-chat-jinja into test-chat-template

* rm test-chat-jinja from cmake

* test-chat-template: use common

* fix build

* fix build (2)

* rename vm --> interpreter

* improve error reporting

* correct lstrip behavior

* add tojson

* more fixes

* disable tests for COMMON_CHAT_FORMAT_GENERIC

* make sure tojson output correct order

* add object.length

* fully functional selectattr / rejectattr

* improve error reporting

* more builtins added, more fixes

* create jinja rendering tests

* fix testing.h path

* adjust whitespace rules

* more fixes

* temporary disable test for ibm-granite

* r/lstrip behavior matched with hf.js

* minimax, glm4.5 ok

* add append and pop

* kimi-k2 ok

* test-chat passed

* fix lstrip_block

* add more jinja tests

* cast to unsigned char

* allow dict key to be numeric

* nemotron: rm windows newline

* tests ok

* fix test

* rename interpreter --> runtime

* fix build

* add more checks

* bring back generic format support

* fix Apertus

* [json.exception.out_of_range.403] key 'content' not found

* rm generic test

* refactor input marking

* add docs

* fix windows build

* clarify error message

* improved tests

* split/rsplit with maxsplit

* non-inverse maxsplit

forgot to change after simplifying

* implement separators for tojson and fix indent

* i like to move it move it

* rename null -- > none

* token::eof

* some nits + comments

* add exception classes for lexer and parser

* null -> none

* rename global -> env

* rm minja

* update docs

* docs: add input marking caveats

* imlement missing jinja-tests functions

* oops

* support trim filter with args, remove bogus to_json reference

* numerous argument fixes

* updated tests

* implement optional strip chars parameter

* use new chars parameter

* float filter also has default

* always leave at least one decimal in float string

* jinja : static analysis + header cleanup + minor fixes

* add fuzz test

* add string.cpp

* fix chat_template_kwargs

* nits

* fix build

* revert

* unrevert

sorry :)

* add fuzz func_args, refactor to be safer

* fix array.map()

* loosen ensure_vals max count condition, add not impl for map(int)

* hopefully fix windows

* check if empty first

* normalize newlines

---------

Co-authored-by: Alde Rojas <hello@alde.dev>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-01-16 11:22:06 +01:00