Ask.Cyberinfrastructure

How do I run mpi4py?

I’m having problems running mpi4py, can anyone help with a basic example? Thank you!

Indeed, mpi4py (and MPI itself) can be remarkably complicated! Although you don’t actually need a scheduler, you at least need a working MPI implementation (like OpenMPI), with an mpi4py that has been built against your MPI implementation[1].

So, providing a simple example Python script is one thing, but you still need a working mpi4py. So, I set up a GitHub repo that has an example Python script, as well as instructions (targeted for Ubuntu bionic) on how to get the example up and running. For people with SLURM environments, I also include an example batch script!

Have a look: https://github.com/akkornel/mpi4py

[1]: “MPI” is not actually a product, it’s a standard. It defines an API (in C and Fortran), for MPI implementations to implement (OpenMPI is one example of an MPI implementation). The code which does MPI stuff (in this case, mpi4py) needs to be built using the MPI implementation’s header files, and run with the MPI implementation’s libraries. Note also that MPI came about before pkg-config was a thing, so MPI implementations provide wrapper scripts (mpicc, etc.) that call the actual tools (like cc) with the appropriate MPI options.

1 Like

It is hard to answer without knowing more details about what problems you are experiencing. It would be helpful to know what OS you are running, if you want to run mpi4py on an HPC system, if you are using a scheduler like SLURM, do you have a working installation of MPI like OpenMPI for example?

Thanks @nrapstine and welcome to AskCI! I was very non-specific / general on purpose, because I wanted to get different scenarios for using it.

Understood. I use mpi4py in two scenarios.

  1. On a supercomputer that already has OpenMPI or MPICH installed. So, I simply load the appropriate mpi module. Then, create/activate a conda environment where I want mpi4py installed and pip install it. Submit a job to a scheduler on a distributed partition that runs my python script. For example, in my slurm script, I would have:

module load mpi python/anaconda3
source activate myenv
srun --mpi=pmi2 python hello_world.py

  1. On my local machine which runs Mac OS. I first installed OpenMPI then pip installed mpi4py in a conda environment and can run the same python script with

source activate myenv
mpirun -np 4 python hello_world.py