We have recently upgraded to using Torque with cgroups. We have been happy with the use of cgroups in general however, I have recently found that MPI process affinity usually does nothing to improve run times and sometimes slows down certain MPI applications. I assume this is due to process affinity interfering with cgroups(?).
My question is, since the MPI versions we support here at W&M (IntelMPI, Mvapich2 and OpenMPI) all have some sort of process binding by default, do you somehow disable MPI process affinity system-wide, do you simply warn your users to investigate the effect of process affinity, or do you not see any effect of processes affinity + cgroups.
Just wondering what the common wisdom is since most google searching and looking at the MPI websites don’t seem to address effects of cgroups and
Thanks for any information that can be shared.