tokenizers
Fast State-of-the-Art Tokenizers optimized for Research and ProductionMore Information
For more information visit the tokenizers website.
Available Versions
These versions of tokenizers are available on the BEAR systems (BlueBEAR and BEAR Cloud VMs). These will be retained in accordance with our Applications Support and Retention Policy.
| Version | BEAR Apps Version |
|---|---|
| 0.19.1-GCCcore-12.3.0 | 2023a |
| 0.15.2-GCCcore-12.3.0 | 2023a |