Need assistance getting DAFoam set up

Q&A
1

Does anyone here have familiarity with DAFoam (DAFoam: Discrete Adjoint with OpenFOAM for High-fidelity Multidisciplinary Design Optimization | DAFoam)?

I’m trying to set up a containerized version using their Docker container. I converted it to Singularity. The container builds, but DAFoam’s recommended approach is to launch a Docker container as the dafoamuser user, whereas Singularity runs containers in the same context as the user and group of who starts the container.

I tried manually sourcing all the files the dafoamuser user would run, but I still get errors when trying to run even the easy parts of a supplied tutorial.

I’m hoping someone out there in “ask.CI-land” has some experience with running this using a singularity container.

Thanks.

2

I didn’t get a response here, but I found a solution through other inquiries. I’ll share it now so that it can help someone else in the future.

We’ll start with the container itself. Be sure to use a current build:

singularity build dafoam_latest.sif docker://dafoam/opt-packages:latest

This will build the Singularity container and save it as dafoam_latest.sif file.

Then the following is put into a SLURM script for processing:

singularity exec dafoam_latest.sif /bin/bash -l -c ‘export HOME=/home/dafoamuser && export MPLCONFIGDIR=/tmp && . /home/dafoamuser/dafoam/loadDAFoam.sh && ./preProcessing.sh && mpirun -np 4 python runScript.py’

The significant part to make this work is the substring:

/bin/bash -l -c 'export HOME=/home/dafoamuser && export MPLCONFIGDIR=/tmp && . /home/dafoamuser/dafoam/loadDAFoam.sh ...

This overrides the default $HOME environment variable and sets it to the “dafoamuser” that the container environment expects.

It then sets expected shell variables via the string:

. /home/dafoamuser/dafoam/loadDAFoam.sh

Everything after, which includes the commands

./preProcessing.sh && mpirun -np 4 python runScript.py

run the tutorial commands preProcessing.sh and runScript.py

these commands are from the tutorial download tutorials-main.tar.gz, with example NACA0012_Airfoil/incompressible:

https://dafoam.readthedocs.io/en/latest/Tutorial_Aerodynamics_NACA0012_Incompressible.html

Note this example used 4 cores (mpirun -np 4)

Because of how containers and openmpi work (or don’t work, as the case may be), this example is limited to a single node.

This is an adaptation of the SLURM script I used, be sure to modify it for your site-specific settings:

#!/bin/bash

#SBATCH --job-name=DAfoam_test # Job name
#SBATCH --output=DAfoam_test.%j.out # Standard output and error file
#SBATCH --nodes=1 # Total number of physical nodes
#SBATCH --ntasks-per-node=4 # Maximum number of tasks on each node
#SBATCH --time=00:10:00 # Job run time limit
#SBATCH --partition=standard # Use standard cores

echo “Job started on hostname at date”

Run DAfoam in its container

singularity exec dafoam_latest.sif /bin/bash -l -c ‘export HOME=/home/dafoamuser && export MPLCONFIGDIR=/tmp && . /home/dafoamuser/dafoam/loadDAFoam.sh && ./preProcessing.sh && mpirun -np 4 python runScript.py’
local_rc=$?

Print status

echo " "
echo “Command exited with return code $local_rc”

Exit

echo " "
echo “Job ended on hostname at date”

Your mileage may vary!

If anyone else has additional information to share, especially getting this to work with MPI across multiple nodes, please follow up!

1 Like