Ask.Cyberinfrastructure

Custom software stack environment for user (no sudo/root)

research-software
administering-hpc

#1

Anybody have experience with tools that can provide user-managed software stack environment for HPC or similar kind of environment (multi-user, multi-node)? (Another scenario is for users given one or more VM’s where he/she could not have root access for any reason.) This needs to be an environment that can be 100% controlled by the user, not by sysadmin.

Background: Under certain circumstances, user may choose to create his/her own software stack. Is there a platform that can be recommended for this user? The user has some basic skill on software install, build, etc but not too high skill in terms of software troubleshooting, etc. Ideally it contains many tested recipes for up-to-date software versions so the work on the user part is not too much.

Constraint: The tool must be able to install to a user-specified location and NOT require superuser privileges (sudo/su) in the setup/build/install/config processes.

I notice there are software distribution with similar philosophy. For example, Conda/Anaconda for Python environment. Then there is also MSYS2, which seems to leverage ArchLinux pacman (?). And cygwin. But for Linux environment, anything like this? I came across Linuxbrew recently. Any input on your experience with this software management tool? How will this compare to Spack, etc?

Wirawan


#2

Conda provides for much more than just python packages. I’ve used conda environments (with miniconda specifically) to create custom software stacks for various projects across different environments, including multiple HPC clusters. Adding channels such as conda-forge and bioconda provides access to a wide variety of packages. And building your own isn’t too hard.


#3

Conda, interestingly, is unable to unload the $PATH variable on deactivate, which makes it a bit inconvenient to switch between environments like modules or lmod does.


#4

“Building your own [package] isn’t too hard”

This was what I was trying to avoid, actually. I added a clarifying stmt in the original question, i.e. what packaging system can be easily deployed by end-users in shared HPC or VM environment? Many HPC/VM users probably don’t have the patience to build a “packaging spec” file (be it for Conda, for LinuxBrew, for Debian, etc.).


#5

@lparsons Can you say more about the packages you used in the past? Does Conda have packages for general (non-python) computing? Say, gcc, gdb, MPI, MKL or OpenBLAS, …?


#6

Can you say more about the packages you used in the past? Does Conda have packages for general (non-python) computing? Say, gcc, gdb, MPI, MKL or OpenBLAS, …?

I’ve used quite a few bioinformatics packages from bioconda, including samtools, STAR, featureCounts, DESeq2, etc. In addition to the biology related bioconda channel, there is an extremely useful channel called conda-forge that is a community driven effort to provide conda recipes for all sorts of general purpose computing tools, including things like OpenBLAS. Finally, there are many other channels, including the Anaconda channel that have many other things available.