tiktoken is a fast BPE tokeniser for use with OpenAI's models. - View it on GitHub
Star
1
Rank
4891825