Spark 3.1.1-foss-2020b

Spark is Hadoop MapReduce done in memory

Accessing Spark 3.1.1-foss-2020b

To load the module for Spark 3.1.1-foss-2020b please use this command on the BEAR systems (BlueBEAR and BEAR Cloud VMs):

📋 module load Spark/3.1.1-foss-2020b

BEAR Apps Version

2020b

Architectures

EL8-cascadelake — EL8-icelake — EL8-sapphirerapids

The listed architectures consist of two part: OS-CPU. The OS used is represented by EL and there are several different processor (CPU) types available on BlueBEAR. More information about the processor types on BlueBEAR is available on the BlueBEAR Job Submission page.

Extensions

py4j 0.10.9.2

More Information

For more information visit the Spark website.

Dependencies

This version of Spark has a direct dependency on: Arrow/0.17.1-foss-2020b foss/2020b Java/11 Python/3.8.6-GCCcore-10.2.0

Required By

This version of Spark is a direct dependent of: hyperopt/0.2.5-foss-2020b

Other Versions

These versions of Spark are available on the BEAR systems (BlueBEAR and BEAR Cloud VMs). These will be retained in accordance with our Applications Support and Retention Policy.

Version	BEAR Apps Version
2.4.4-foss-2019a-Hadoop-2.7-Java-11-Python-3.7.2	2019a
2.4.1-foss-2019b-Python-3.7.4-Java-1.8	2019b

Last modified on 21st May 2021