tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models

More Information

For more information visit the tiktoken website.

Available Versions

These versions of tiktoken are available on the BEAR systems (BlueBEAR and BEAR Cloud VMs). These will be retained in accordance with our Applications Support and Retention Policy.