Spark 3.1.1-fosscuda-2020bSpark is Hadoop MapReduce done in memory
Accessing Spark 3.1.1-fosscuda-2020b
To load the module for Spark 3.1.1-fosscuda-2020b please use this command on the BEAR systems (BlueBEAR, BEARCloud VMs, and CaStLeS VMs):
module load Spark/3.1.1-fosscuda-2020b
There is a CPU version of this module: Spark 3.1.1-foss-2020b
BEAR Apps Version
EL8-haswell (GPUs: NVIDIA P100)
The listed architectures consist of two part: OS-CPU.
- BlueBEAR: The OS used on BlueBEAR is represented by EL and there are several different processor (CPU) types available on BlueBEAR. More information about the processor types on BlueBEAR is available on the BlueBEAR Job Submission page.
- BEAR and CaStLeS Cloud VMs: These VMs can have one of two OSes. Those with access to a BEAR Cloud or CaStLeS VM should check that the listed architectures for an application include the OS of VM being used. The VMs, irrespective of OS, will use the haswell CPU type.
- py4j 0.10.9.2
For more information visit the Spark website.
This version of Spark has a direct dependency on: Arrow/0.17.1-fosscuda-2020b fosscuda/2020b Java/11 Python/3.8.6-GCCcore-10.2.0
This version of Spark is a direct dependent of: hyperopt/0.2.5-fosscuda-2020b
These versions of Spark are available on the BEAR systems (BlueBEAR, BEARCloud VMs, and CaStLeS VMs). These will be retained in accordance with our Applications Support and Retention Policy.
|Version||BEAR Apps Version|
Last modified on 28th July 2021