# What is the most efficient Python package to run a parallel job over multiple nodes?

**URL:** <https://ask.cyberinfrastructure.org/t/what-is-the-most-efficient-python-package-to-run-a-parallel-job-over-multiple-nodes/171>\
**Category:** Q&A\
**Tags:** parallelization, programming-for-hpc, python, researcher\
**Created:** [April 13, 2018, 2:58pm UTC](https://ask.cyberinfrastructure.org/t/what-is-the-most-efficient-python-package-to-run-a-parallel-job-over-multiple-nodes/171 "2018-04-13T14:58:46Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![jpessin1](https://ask.cyberinfrastructure.org/user_avatar/ask.cyberinfrastructure.org/jpessin1/32/339_2.png) [@jpessin1](https://ask.cyberinfrastructure.org/u/jpessin1)\
**Post date:** [May 25, 2018, 10:14pm UTC](https://ask.cyberinfrastructure.org/t/what-is-the-most-efficient-python-package-to-run-a-parallel-job-over-multiple-nodes/171/2 "2018-05-25T22:14:16Z")

</div>

If you are looking for parallel processing the traditional (and still very valid) approach is use an MPI library,  
mpi4py is an example of a python based wrapper, and [https://mpi4py.readthedocs.io/en/stable/intro.html](https://mpi4py.readthedocs.io/en/stable/intro.html)  
includes a good overview of the concepts and related methods. (Not an endorsement, just not reinventing the wheel here)

Some other things to consider:  
Would it be less work to make the job fit on a single node? With tools like concurrent.futures (or the underlying multiprocessing & threading modules) or mixed tools like numpy/scipy/pandas with Cython?

_Would a faster python implementation (like pypy) provide enough speed?_

Not that they go away when you move to multi-node, but they are often, though not always sufficient and less demanding of the user/developers time.

---

_[View the full topic](https://ask.cyberinfrastructure.org/t/what-is-the-most-efficient-python-package-to-run-a-parallel-job-over-multiple-nodes/171)._