Ask.Cyberinfrastructure

Running R code in parallel in an HPC environment

zeta:se-ready
r
parallelization
programming-for-hpc

#1

Which R packages can I use to parallelize my code on an HPC cluster? Are there any packages that allow parallelization over multiple nodes?

CURATOR: Katia


#2

ANSWER:

I am not very familiar with them, but there are a few packages in R which can do parallelization using MPI, and therefore should be able to parallelize over multiple nodes.

Rmpi I believe is the package which provides the underlying, low-level MPI support for R.

However, I believe most people find the snow package to be higher level/more user friendly. I believe snow leverages Rmpi to allow parallelization across nodes, but might also be usable without Rmpi (i.e. using OpenMP for parallelization only within a node). There is also a doSNOW package which builds on snow and provides a parallel loop function.

When using snow with MPI, a couple of potential gotchas:

  1. The mpirun command for snow based R codes should typically use -np 1 — typically snow spawns its own workers. You still need to tell the scheduler to allocate the correct number of cores.

  2. Most snow based R code will at some point invoke the makeCluster function, which takes a parameter indicating the size of the “cluster” to create. Usually, one wants this to be one less than the number of cores requested from the scheduler, as the main task, which spawns the other tasks, is already consuming one core.

I.e., if you are telling makeCluster to create a cluster of 100 workers,
you should request 101 cores from Slurm or whatever scheduler you are using.
If by mistake you request the same number (e.g. 100) from both the scheduler
and the makeCluster command, there will be one core oversubscribed which
generally will cause issues. Typically, I see the R script reporting an
error about an insufficient number of “slots” being available, and just hanging
(doing nothing but not dying til the job is killed).


#3

The relevant R packages are listed and described in the HPC task view, available at https://cran.r-project.org/web/views/HighPerformanceComputing.html. You can contribute to this list at https://github.com/eddelbuettel/ctv-hpc if you find something missing or incomplete.