I am interested in knowing which of the ACCESS systems allow for multinode distributed GPU applications (e.g., I have 2 compute nodes each with 4 gpus for a total of 8 gpus). I am aware that Delta has this capability but in documentation of other systems it is not clear to me. Thank you!
1 Like
I don’t understand what you mean by capability. My understanding of computation on multi-nodes, is that there should be the communication piece between the nodes and the application being executed should have been programmed to do that. Programing paradigm in GPU is essentially to handle the computation. The communication between nodes is really independent of the type of processing units.
I could be wrong but will love to read what others think
Let me clarify my question. The SCC at Boston University does not allow you to request and use GPUs on 2 different nodes. You are only able to use a single node when you request GPUs.
I understand that hypothetically as long as nodes can communicate this should be possible. Essentially I am interested in knowing if there restrictions in place to prevent such communication between nodes on other ACCESS systems.
The effective way is to not install MPI on the nodes and that is it
@jfossot Let me clarify the question: Which ACCESS resources can run code that requires multiple GPUs (say 20 or 40). Currently, most compute nodes have 2 or 4 GPUs each, so to be able to scale the code to use 20 or 40 GPUs, one needs to request multiple nodes. The code is written to support it and it runs fine on Delta. The question is - what other ACCESS resources that can run this code?
Thank you for the clarification. I am looking at the ACCESS resource site and on the “filter by feature” and don’t see anything implying distributed memory jobs (by default they are all shared memory). But if a system has MPI set up, it should be able to do distributed memory jobs.
I am curious as to why you will need multiple GPU nodes. If I am the system administrator, I will not allow that unless you show me performance data to justify the need for many GPUs. A GPU by itself has a lot of compute power to do most of the common task, if the program is well optimized.
1 Like
Thank you @jfossot . The link you provided is very helpful! It does not answer this particular question (multinode GPU jobs), but it helps explore ACCESS resources for some other types of jobs. Thank you!
As for your “system administrator” question, I agree - scaling the code to take advantage of multiple GPUs (using MPI) is tricky. However, there are more and more software packages that allow it ( some were featured on the last PEARC conference).
Some relevant links:
@sladenheim @ktrn catching this a few months late. The Bridges-2 GPU partition at PSC supports multi-node GPU jobs. You may also want to check SDSC Expanse since they’re similar systems.
1 Like
Do you know happen to know the max number of nodes/gpus you could request for the Bridges-2 GPU partition?
@sladenheim I do!
In theory, you could request every node and every GPU. In practice, it would never get through slurm. The answer largely depends on “how many nodes do you need?”
From October 1, 2022 - September 30, 2023 over 450,000 GPU jobs were ran on Bridges-2. Only 711 of those scaled beyond a single node of 8 GPUs.
Bridges-2 has 33 GPU nodes (8 GPUs per node) + 1 Nvidia DGX-2 (16 GPUs). If you’re interested in scaling across a large number of GPUs, Bridges-2 might not be the best resource for this as resources like Anvil and Delta may have better economies of scale.
If you already have an account on Bridges-2, or if you’re interested in one, I strongly encourage contacting help@psc.edu with a little more detail on your needs and the support team there will give you their take on your request.
Best wishes!
Stephen