tokenizers 0.19.1-GCCcore-12.3.0

Fast State-of-the-Art Tokenizers optimized for Research and Production

Accessing tokenizers 0.19.1-GCCcore-12.3.0

To load the module for tokenizers 0.19.1-GCCcore-12.3.0 please use this command on the BEAR systems (BlueBEAR and BEAR Cloud VMs):

📋 module load bear-apps/2023a
module load tokenizers/0.19.1-GCCcore-12.3.0

BEAR Apps Version

2023a

Architectures

EL8-icelake — EL8-sapphirerapids

The listed architectures consist of two parts: OS-CPU. The OS used is represented by EL and there are several different processor (CPU) types available on BlueBEAR. More information about the processor types on BlueBEAR is available on the BlueBEAR Job Submission page.

Extensions

• huggingface-hub-0.24.5
• tokenizers 0.19.1

More Information

For more information visit the tokenizers website.

Dependencies

This version of tokenizers has a direct dependency on: GCCcore/12.3.0 Python/3.11.3-GCCcore-12.3.0 Python-bundle-PyPI/2023.06-GCCcore-12.3.0 PyYAML/6.0-GCCcore-12.3.0 tqdm/4.66.1-GCCcore-12.3.0

Required By

This version of tokenizers is a direct dependent of: Transformers/4.42.0-foss-2023a-CUDA-12.1.1 Transformers/4.42.0-foss-2023a

Other Versions

These versions of tokenizers are available on the BEAR systems (BlueBEAR and BEAR Cloud VMs). These will be retained in accordance with our Applications Support and Retention Policy.

Last modified on 2nd April 2025