# My R loop slows down when I increase the number of iterations on an HPC

**URL:** <https://ask.cyberinfrastructure.org/t/my-r-loop-slows-down-when-i-increase-the-number-of-iterations-on-an-hpc/215>\
**Category:** Q&A\
**Tags:** programming-for-hpc, r, scaling, researcher\
**Created:** [May 31, 2018, 8:56pm UTC](https://ask.cyberinfrastructure.org/t/my-r-loop-slows-down-when-i-increase-the-number-of-iterations-on-an-hpc/215 "2018-05-31T20:56:36Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ktrn](https://ask.cyberinfrastructure.org/user_avatar/ask.cyberinfrastructure.org/ktrn/32/554_2.png) [@ktrn](https://ask.cyberinfrastructure.org/u/ktrn)\
**Post date:** [May 31, 2018, 8:56pm UTC](https://ask.cyberinfrastructure.org/t/my-r-loop-slows-down-when-i-increase-the-number-of-iterations-on-an-hpc/215/1 "2018-05-31T20:56:36Z")

</div>

I wrote an R script and tested it on my laptop. It finishes 100 iterations in about 1 minute. However I need to run this code for 10,000 iterations, so I submitted my job to run on a cluster. I expected my script to take about 100 times longer (since I use 100 times more iterations), but it looks like it runs much slower than on my local computer. Why? Here is a script I am using:

```
x <- matrix(rnorm(10000000), nrow=10000)
x[sample(1:10000000, 10000, replace=FALSE)] <- NA

  # initialize res 
  res = NULL
  n = nrow(x)
  
  for (i in 1:n) {
    if ( sum(is.na(x[i,])) < 3 ) res = rbind(res, x[i,])
  }

  apply(res,1,mean)

```

---

<div class="post-metadata">

**Author:** ![ktrn](https://ask.cyberinfrastructure.org/user_avatar/ask.cyberinfrastructure.org/ktrn/32/554_2.png) [@ktrn](https://ask.cyberinfrastructure.org/u/ktrn)\
**Post date:** [May 31, 2018, 9:09pm UTC](https://ask.cyberinfrastructure.org/t/my-r-loop-slows-down-when-i-increase-the-number-of-iterations-on-an-hpc/215/2 "2018-05-31T21:09:27Z")

</div>

rbind() function is very inefficient when used within a large loop.  
Your code dynamically re-allocates (re-addresses) memory in each iteration of the loop. A better approach is to pre-allocate the memory for the final size.  
Here you start with an empty matrix and you ”grow” it inside  
the loop. During each iteration of the loop, the computer erases the previous matrix with  
i-1 rows and creates a new matrix with i rows and copies the previous values into it. So with larger values of i this operation is slower and slower.

Try the following:

```
  res = x # initialize res with the size of x
  k = 0
  for (i in 1:n) {
    if (sum(is.na(x[i,]) )< 3) {
      res[k,] = x[i,]
      k = k + 1
    }
  }
  rowMeans(res[1:k,], na.rm=TRUE)

```