My R loop slows down when I increase the number of iterations on an HPC

r
programming-for-hpc

#1

I wrote an R script and tested it on my laptop. It finishes 100 iterations in about 1 minute. However I need to run this code for 10,000 iterations, so I submitted my job to run on a cluster. I expected my script to take about 100 times longer (since I use 100 times more iterations), but it looks like it runs much slower than on my local computer. Why? Here is a script I am using:

x <- matrix(rnorm(10000000), nrow=10000)
x[sample(1:10000000, 10000, replace=FALSE)] <- NA

  # initialize res 
  res = NULL
  n = nrow(x)
  
  for (i in 1:n) {
    if ( sum(is.na(x[i,])) < 3 ) res = rbind(res, x[i,])
  }

  apply(res,1,mean)

#2

rbind() function is very inefficient when used within a large loop.
Your code dynamically re-allocates (re-addresses) memory in each iteration of the loop. A better approach is to pre-allocate the memory for the final size.
Here you start with an empty matrix and you ”grow” it inside
the loop. During each iteration of the loop, the computer erases the previous matrix with
i-1 rows and creates a new matrix with i rows and copies the previous values into it. So with larger values of i this operation is slower and slower.

Try the following:

  res = x # initialize res with the size of x
  k = 0
  for (i in 1:n) {
    if (sum(is.na(x[i,]) )< 3) {
      res[k, ] = x[i,]
      k = k + 1
    }
  }
  rowMeans(res[1:k,], na.rm=TRUE)