vocab: add gemma4 tokenizer tests, fix edge case (#21534)

Browse Source

* YATF (Yet Another Tokenizer Fix) for Gemma 4. With tests!
* Remove unnecessary hash  from update script.
* minor: move constant

...

This commit is contained in:

Piotr Wilkin (ilintar)

2026-04-09 11:41:14 +02:00

committed by GitHub

parent 243532e556

commit 0ec191e1d7

5 changed files with 169 additions and 2 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

models/ggml-vocab-gemma-4.gguf

BIN

Binary file not shown.