Ask.Cyberinfrastructure

Using LMOD to load Python3 libraries from a user's home directory

python
lmod
environment-modules

#1

I’ve installed a local Python3 library at /home/$USER/python3lib. How can I use LMOD to load this library?

CURATOR: Grace Wilson Caudill/Scott Valcourt


#4

I think what you might also be asking is how to write a modulefile so that your python program can be loaded with LMOD? Let’s walk through the steps, and the full guide is available here.

Write your File

The file itself is in a language called “lua” and for most, you really just need to prepend to the path to make an executable seen. Here is a simple example: I’m assuming that when you say pyhthon3 library you mean just a module, so we need it seen by $PYTHONPATH (and not a full python installation).

help([[
     Add python modules from /opt/path
]])
whatis("Keywords: System, Python")
whatis("Description: Python, Modules")

prepend_path( "PYTHONPATH",           "/opt/path")

I think for an environment variable you may need to do something like:

local username = os.getenv("USER")
local pythonDir  = pathJoin("/home", username, "pyrhon3lib")
prepend_path("PYTHONPATH", pythonDir)

If your module is in some version folder, don’t forget to add that to the path you prepend! Take note there are also functions to setenv

setenv("VARIABLE", "value")

And these are unset when you unload a module. There is also a useful function to append to the path:

append_path("PATH", "/opt/app/bin")

Check out all the functions here.

Put it Somewhere

If you are a user adding the file, you can put it anywhere really, then in your .profile add a command to use it:

module use $HOME/modules/pythonlib/1.0

If you are an admin, you probably already know how to do this, but $MODULEPATH includes the paths that LMOD can see, so you can add your module there, or still do the above for a custom location (the command above adds it to the same path). For example, here is $MODULEPATH on one of our clusters:

$ echo $MODULEPATH
/share/software/modules/system:/share/software/modules/math:/share/software/modules/devel:/share/software/modules/categories

If we peek into a folder, we see the organization of modules (this is just a subset at the top level)-

$ tree /share/software/modules/
biology/         physics/       /categories

But how does LMOD know when to look inside physics? This probably varies based on how the resource has it set up (see the docs I’ve linked to read more) but at least for our cluster we have a categories folder (likely added somewhere, I’m not sure where), again with lua files that match each folder at the root, eg:

$ tree /share/software/modules/categories
/share/software/modules/categories
├── biology.lua -> .template
├── chemistry.lua -> .template
├── devel.lua -> .template_sticky
├── labs.lua -> .template
├── math.lua -> .template_sticky
├── physics.lua -> .template
├── README
├── staging.lua
├── system.lua -> .template
└── viz.lua -> .template

and this is the content of a file, which is actually a link to a template, and the only difference is that the name of the file will determine myModuleName() and thus add the complete path in the root folder (e.g., physics)

local mroot = os.getenv("MODULEPATH_ROOT")
local mdir  = pathJoin(mroot,myModuleName())
prepend_path("MODULEPATH", mdir)

And then some physics lua file (for example /share/software/modules/physics/opensees/2.5.0.lua named by version) knows it’s category because:

...
whatis("Category: physics")

And I’d say it might be good practice to keep this under version control. Now if we are to peek into the physics subfolder, we see a particular organization - did you notice how I used a format of <software-name>/<version> ? That’s done so your future users can load multiple versions, e.g.,

module load pythonlib/1.0.0
module load pythonlib/1.2.0

And it also means you can automatically determine the version and name from the path:

whatis("Name:        ", myModuleName())
whatis("Version:     ", myModuleVersion())

There is an entire section on that too.

Test and Learn More

Anyway, there are likely many ways to go about this, and I know so few of the tricks that I’ll stop there! I think your best bet is to keep it simple, and then when you want advanced functionality look to the documentation and other examples online. Finally, a great place to learn is just logging into your clusters, looking at $MODULEPATH, and then inspecting the contents of the folder. Definitely test your module, when you think it’s good:

module load pythonlib3

And then as @sav mentioned, you should be able to search with module avail and module spider <term> It’s really not so bad if it doesn’t work perfectly the first time, just fix up the file (try something different) and try again! You really can’t fail at these things, it comes down to a fancy way for adding/removing from various paths.


#2

To load a local Python3 library module at /home/$USER/python3lib using LMOD, first determine that the module was loaded and is available by entering:

$ module avail

If the module appears in the listing, presuming that the module name is python3lib, enter the following command at the command prompt:

$ module load python3lib

#3

$ module avail will show a list of all modules available for loading on the given HPC system. To see if the module in question is already loaded, use the $ module list command. This command will tell you which modules are already in your environment.