tokeni.zig (stands for tokenizer + Zig) is a minimal implementation of a Byte Pair Encoding (BPE) tokenizer in Zig - View it on GitHub