# \#qow **URL:** https://ask.cyberinfrastructure.org/tag/qow/175.md [Latest](https://ask.cyberinfrastructure.org/latest.md) · [Categories](https://ask.cyberinfrastructure.org/categories.md) · [Tags](https://ask.cyberinfrastructure.org/tags.md) --- ## [How do I create my own cluster?](https://ask.cyberinfrastructure.org/t/how-do-i-create-my-own-cluster/856)
**Author:** [@vsoch](https://ask.cyberinfrastructure.org/u/vsoch)\ **Replies:** 5\ **Last updated:** [May 23, 2024, 5:35pm UTC](https://ask.cyberinfrastructure.org/t/how-do-i-create-my-own-cluster/856 "2024-05-23T17:35:17Z")
I’ve been seeing posts around like this and I’m genuinely interested in what it takes to create your own little cluster. A lot of these discussions start with the assumption that I have a crapton of old equipment lying a… --- ## [Using anaconda.org to find software (Promoted Locale Entry)](https://ask.cyberinfrastructure.org/t/using-anaconda-org-to-find-software-promoted-locale-entry/1970)
**Author:** [@jma](https://ask.cyberinfrastructure.org/u/jma)\ **Replies:** 1\ **Last updated:** [May 23, 2024, 5:23pm UTC](https://ask.cyberinfrastructure.org/t/using-anaconda-org-to-find-software-promoted-locale-entry/1970 "2024-05-23T17:23:47Z")
The NAU team posted this excellent write up of how they use Anaconda to find software. What does your institution recommend? (Click on the caret in the top right corner to read the post inline and the post your reply her… --- ## [Multi-instance GPU](https://ask.cyberinfrastructure.org/t/multi-instance-gpu/1710)
**Author:** [@wirawan0](https://ask.cyberinfrastructure.org/u/wirawan0)\ **Replies:** 2\ **Last updated:** [October 1, 2023, 2:05am UTC](https://ask.cyberinfrastructure.org/t/multi-instance-gpu/1710 "2023-10-01T02:05:53Z")
Hi folks, the new Ampere A100 GPU supports “multi-instance GPU”, i.e. allowing a GPU to be partitioned so multiple people can use the GPU to maximize throughout of computations. Other than Ampere A100, any other cards or… --- ## [Optimizing very large neural network that is greater than size of GPU memory](https://ask.cyberinfrastructure.org/t/optimizing-very-large-neural-network-that-is-greater-than-size-of-gpu-memory/427)
**Author:** [@wirawan0](https://ask.cyberinfrastructure.org/u/wirawan0)\ **Replies:** 3\ **Last updated:** [August 17, 2023, 3:10pm UTC](https://ask.cyberinfrastructure.org/t/optimizing-very-large-neural-network-that-is-greater-than-size-of-gpu-memory/427 "2023-08-17T15:10:30Z")
I encountered a problem with extremely large neural network that was created in KERAS, using Tensorflow backend. The memory footprint in one of the layer is already bigger than the size of current GPU memory (it has just… --- ## [IoT Devices and VPN](https://ask.cyberinfrastructure.org/t/iot-devices-and-vpn/1826)
**Author:** [@raygun](https://ask.cyberinfrastructure.org/u/raygun)\ **Replies:** 1\ **Last updated:** [May 22, 2023, 8:46pm UTC](https://ask.cyberinfrastructure.org/t/iot-devices-and-vpn/1826 "2023-05-22T20:46:49Z")
Some of our researchers are conducting IoT based research where their IoT devices are located off campus but their servers are on campus. Currently, most connections to university-wide systems require VPN. However, to th… --- ## [What qualifies as a virtual environment, and when is it appropriate to create a virtual environment?](https://ask.cyberinfrastructure.org/t/what-qualifies-as-a-virtual-environment-and-when-is-it-appropriate-to-create-a-virtual-environment/2246)
**Author:** [@toreliza](https://ask.cyberinfrastructure.org/u/toreliza)\ **Replies:** 4\ **Last updated:** [May 12, 2023, 2:48pm UTC](https://ask.cyberinfrastructure.org/t/what-qualifies-as-a-virtual-environment-and-when-is-it-appropriate-to-create-a-virtual-environment/2246 "2023-05-12T14:48:08Z")
Requests to set up virtual environments (python, containers, os-related, for example) seem to be increasing in frequency. A workshop in which I recently participated required attendees to create a virtual environment fo… --- ## [How is storage performance for high I/O HPC jobs affected by running in the cloud?](https://ask.cyberinfrastructure.org/t/how-is-storage-performance-for-high-i-o-hpc-jobs-affected-by-running-in-the-cloud/797)
**Author:** [@toreliza](https://ask.cyberinfrastructure.org/u/toreliza)\ **Replies:** 3\ **Last updated:** [March 16, 2023, 5:56am UTC](https://ask.cyberinfrastructure.org/t/how-is-storage-performance-for-high-i-o-hpc-jobs-affected-by-running-in-the-cloud/797 "2023-03-16T05:56:33Z")
We have some projects that include running calculations with high I/O on our university HPC cluster. We are considering moving them to cloud, probably AWS. I believe that the data to be used will reside in a data cente… --- ## [DUO for Multi-Factor Authentication on HPC Systems](https://ask.cyberinfrastructure.org/t/duo-for-multi-factor-authentication-on-hpc-systems/268)
**Author:** [@toreliza](https://ask.cyberinfrastructure.org/u/toreliza)\ **Replies:** 10\ **Last updated:** [March 14, 2023, 6:45pm UTC](https://ask.cyberinfrastructure.org/t/duo-for-multi-factor-authentication-on-hpc-systems/268 "2023-03-14T18:45:25Z")
We’re planning to implement multi-factor authentication (DUO specifically) on our HPC systems. It seems like it should be relatively straight-forward if user authentication at login (via the head/login nodes) is the onl… --- ## [Red Hat and CentOS Linux 8 Status](https://ask.cyberinfrastructure.org/t/red-hat-and-centos-linux-8-status/2370)
**Author:** [@toreliza](https://ask.cyberinfrastructure.org/u/toreliza)\ **Replies:** 4\ **Last updated:** [February 15, 2023, 12:27am UTC](https://ask.cyberinfrastructure.org/t/red-hat-and-centos-linux-8-status/2370 "2023-02-15T00:27:15Z")
What decisions have been made, and what subsequent experience can institutions share concerning the Red Hat discontinuation of CentOS Linux 8 as a rebuild of RHEL 8? Is there one dominant direction? Or a clear advantag… --- ## [Bursting to Cloud from On-Prem HPC Cluster](https://ask.cyberinfrastructure.org/t/bursting-to-cloud-from-on-prem-hpc-cluster/2205)
**Author:** [@j\_fossot](https://ask.cyberinfrastructure.org/u/j_fossot)\ **Replies:** 3\ **Last updated:** [January 11, 2023, 1:49pm UTC](https://ask.cyberinfrastructure.org/t/bursting-to-cloud-from-on-prem-hpc-cluster/2205 "2023-01-11T13:49:51Z")
When is it appropriate to burst to cloud from an on-prem HPC cluster? Are there certain metrics and ranges for those metrics that might serve as triggers? What factors go into a decision to ‘burst to cloud’? --- ## [Cloud native technologies and innovation](https://ask.cyberinfrastructure.org/t/cloud-native-technologies-and-innovation/2172)
**Author:** [@vsoch](https://ask.cyberinfrastructure.org/u/vsoch)\ **Replies:** 2\ **Last updated:** [December 24, 2022, 11:56am UTC](https://ask.cyberinfrastructure.org/t/cloud-native-technologies-and-innovation/2172 "2022-12-24T11:56:09Z")
We’ve been thinking more about how to create hybrid environments and better bridge the gap between HPC, cloud, and local, and to support more modern workflows for researchers (e.g., think APIs, services, and automation). … --- ## [Running COMSOL with MATLAB using LiveLink on SLURM cluster](https://ask.cyberinfrastructure.org/t/running-comsol-with-matlab-using-livelink-on-slurm-cluster/281)
**Author:** [@toreliza](https://ask.cyberinfrastructure.org/u/toreliza)\ **Replies:** 0\ **Last updated:** [July 12, 2018, 10:25pm UTC](https://ask.cyberinfrastructure.org/t/running-comsol-with-matlab-using-livelink-on-slurm-cluster/281 "2018-07-12T22:25:25Z")
A few years ago, I was working with a researcher trying to run a calculation in Matlab while executing a job in COMSOL, and to use the results of the Matlab calculation in the running COMSOL job. Much effort was put int… --- ## [Data Transfer with GoAnywhere?](https://ask.cyberinfrastructure.org/t/data-transfer-with-goanywhere/1131)
**Author:** [@ckravit](https://ask.cyberinfrastructure.org/u/ckravit)\ **Replies:** 0\ **Last updated:** [November 13, 2019, 10:49pm UTC](https://ask.cyberinfrastructure.org/t/data-transfer-with-goanywhere/1131 "2019-11-13T22:49:53Z")
Has anyone here used the GoAnywhere software for data transfer? How about their GoFast “accelerated” software? We’re considering it as an alternative to Globus and I’m wondering if how much use it has in the community a… --- ## [Monitoring GPU usage](https://ask.cyberinfrastructure.org/t/monitoring-gpu-usage/932)
**Author:** [@jma](https://ask.cyberinfrastructure.org/u/jma)\ **Replies:** 4\ **Last updated:** [December 6, 2021, 8:37pm UTC](https://ask.cyberinfrastructure.org/t/monitoring-gpu-usage/932 "2021-12-06T20:37:28Z")
I’m running a calculation that parallelizes well using both MPI and OpenMP. While reading online, it occurred to me that I might gain speedup with a GPU node. How can I estimate the potential benefit? And is it possib… --- ## [CPU binding: What are some appropriate uses?](https://ask.cyberinfrastructure.org/t/cpu-binding-what-are-some-appropriate-uses/2137)
**Author:** [@toreliza](https://ask.cyberinfrastructure.org/u/toreliza)\ **Replies:** 1\ **Last updated:** [November 30, 2021, 4:39pm UTC](https://ask.cyberinfrastructure.org/t/cpu-binding-what-are-some-appropriate-uses/2137 "2021-11-30T16:39:33Z")
Some colleagues were debating the merits of scheduler binding options, and we realized we lacked clarity as to when binding tasks to resources might be most beneficial. Below is a specific topic: On most clusters, one … --- ## [How do I use Anaconda to install & manage software on Monsoon?](https://ask.cyberinfrastructure.org/t/how-do-i-use-anaconda-to-install-manage-software-on-monsoon/1486)
**Author:** [@jasonbuechler](https://ask.cyberinfrastructure.org/u/jasonbuechler)\ **Replies:** 0\ **Last updated:** [August 26, 2020, 11:56pm UTC](https://ask.cyberinfrastructure.org/t/how-do-i-use-anaconda-to-install-manage-software-on-monsoon/1486 "2020-08-26T23:56:35Z")
Running Anaconda on Monsoon Monsoon offers several versions of Anaconda through our software module system. Since Anaconda is tightly integrated with Python, available versions are specifically divided into forks that pr… --- ## [Overwriting labels during singularity container build](https://ask.cyberinfrastructure.org/t/overwriting-labels-during-singularity-container-build/1881)
**Author:** [@wirawan0](https://ask.cyberinfrastructure.org/u/wirawan0)\ **Replies:** 1\ **Last updated:** [July 13, 2021, 3:54pm UTC](https://ask.cyberinfrastructure.org/t/overwriting-labels-during-singularity-container-build/1881 "2021-07-13T15:54:55Z")
In the course of building a Singularity container, I am building a container (call it “B”) based on another container (“A”). In the “B” container def file, I have specified labels to update the values of the labels set i… --- ## [Slurm, GPU, CGroups, ConstrainDevices](https://ask.cyberinfrastructure.org/t/slurm-gpu-cgroups-constraindevices/1745)
**Author:** [@stryder](https://ask.cyberinfrastructure.org/u/stryder)\ **Replies:** 2\ **Last updated:** [May 11, 2021, 9:47pm UTC](https://ask.cyberinfrastructure.org/t/slurm-gpu-cgroups-constraindevices/1745 "2021-05-11T21:47:29Z")
Has any site managed to make slurm + constraindevices + cgroups work well enough that it actually prevents people from manually setting CUDA\_VISIBLE\_DEVICES and double booking processes on GPU? I can see constraindevice… --- ## [Cluster Monitoring Tools](https://ask.cyberinfrastructure.org/t/cluster-monitoring-tools/1793)
**Author:** [@shmget](https://ask.cyberinfrastructure.org/u/shmget)\ **Replies:** 4\ **Last updated:** [May 11, 2021, 3:48pm UTC](https://ask.cyberinfrastructure.org/t/cluster-monitoring-tools/1793 "2021-05-11T15:48:07Z")
We’ve decided to re-evaluate our cluster monitoring tools, and are wondering what current (and new tools) are out there. For instance, there are tools that have been around for some time, like Nagios. There are other too… --- ## [Milan/Ice Lake topology and cooling](https://ask.cyberinfrastructure.org/t/milan-ice-lake-topology-and-cooling/1837)
**Author:** [@Ben](https://ask.cyberinfrastructure.org/u/Ben)\ **Replies:** 0\ **Last updated:** [April 7, 2021, 5:44am UTC](https://ask.cyberinfrastructure.org/t/milan-ice-lake-topology-and-cooling/1837 "2021-04-07T05:44:25Z")
Hi, Wondering which will be the preferred Milan/Ice Lake SKUs this generation for HPC/HTC/scientific computing? Are there any issues with topology within the socket, boundaries where the internal latencies increase? W… --- ## [What resources are used in academia to publish apps](https://ask.cyberinfrastructure.org/t/what-resources-are-used-in-academia-to-publish-apps/1831)
**Author:** [@ktrn](https://ask.cyberinfrastructure.org/u/ktrn)\ **Replies:** 1\ **Last updated:** [April 12, 2021, 8:46pm UTC](https://ask.cyberinfrastructure.org/t/what-resources-are-used-in-academia-to-publish-apps/1831 "2021-04-12T20:46:20Z")
We have more and more researchers who would like to publish an app (to be used worldwide), that requires a significant amount of memory, computational resources (multiple cores, GPU, etc. ). Some examples include R Shiny… --- ## [CP2K Build on CentOS 7](https://ask.cyberinfrastructure.org/t/cp2k-build-on-centos-7/1726)
**Author:** [@ndanes](https://ask.cyberinfrastructure.org/u/ndanes)\ **Replies:** 2\ **Last updated:** [February 15, 2021, 5:12pm UTC](https://ask.cyberinfrastructure.org/t/cp2k-build-on-centos-7/1726 "2021-02-15T17:12:48Z")
Hello! I have been struggling to build CP2K 7.1 on our clusters on one of our HPC clusters and I was wondering if people have had success with builds, depending on their toolchain. Without getting into incredible detail… --- ## [Slurm: Gres vs --gpus configuration, syntax preference on A100](https://ask.cyberinfrastructure.org/t/slurm-gres-vs-gpus-configuration-syntax-preference-on-a100/1652)
**Author:** [@stryder](https://ask.cyberinfrastructure.org/u/stryder)\ **Replies:** 1\ **Last updated:** [November 30, 2020, 7:42pm UTC](https://ask.cyberinfrastructure.org/t/slurm-gres-vs-gpus-configuration-syntax-preference-on-a100/1652 "2020-11-30T19:42:22Z")
Previously many sites used the GRES syntax for user facing commands to request GPU using slurm. Now with the newer --gpus syntax is there a preference? Can the two co-exist at the same time for users bringing older codes… --- ## [Going between XML (or JSON) and table-like formats](https://ask.cyberinfrastructure.org/t/going-between-xml-or-json-and-table-like-formats/1547)
**Author:** [@jpessin1](https://ask.cyberinfrastructure.org/u/jpessin1)\ **Replies:** 3\ **Last updated:** [November 17, 2020, 4:42pm UTC](https://ask.cyberinfrastructure.org/t/going-between-xml-or-json-and-table-like-formats/1547 "2020-11-17T16:42:31Z")
Are there any good SQL or other table based methods to support the variability in XML/JSON data? When is there a standard way to flatten this for tabular work? As a simple example illustration, NCBI database are access… --- ## [What is a checkpoint?](https://ask.cyberinfrastructure.org/t/what-is-a-checkpoint/1021)
**Author:** [@paulobaptista](https://ask.cyberinfrastructure.org/u/paulobaptista)\ **Replies:** 2\ **Last updated:** [August 17, 2020, 7:44pm UTC](https://ask.cyberinfrastructure.org/t/what-is-a-checkpoint/1021 "2020-08-17T19:44:48Z")
I was watching a video about a HPC facility in Canada talk about their new storage solution. They mentioned that checkpoints used to run in the hours, but with the new solution it takes 1/2 hour. In the HPC world, what… --- ## [Softwares to support Social Scientists](https://ask.cyberinfrastructure.org/t/softwares-to-support-social-scientists/1325)
**Author:** [@schadalapaka](https://ask.cyberinfrastructure.org/u/schadalapaka)\ **Replies:** 2\ **Last updated:** [July 30, 2020, 2:21pm UTC](https://ask.cyberinfrastructure.org/t/softwares-to-support-social-scientists/1325 "2020-07-30T14:21:15Z")
Hi, Hope you’re all doing well and are keeping safe. At this point, many of our Social Scientists are exploring ways to move some of their projects online. What softwares for this purpose do your campuses support? The… --- ## [Automated HPC testing tools and techniques?](https://ask.cyberinfrastructure.org/t/automated-hpc-testing-tools-and-techniques/961)
**Author:** [@mrich](https://ask.cyberinfrastructure.org/u/mrich)\ **Replies:** 9\ **Last updated:** [July 11, 2020, 4:10am UTC](https://ask.cyberinfrastructure.org/t/automated-hpc-testing-tools-and-techniques/961 "2020-07-11T04:10:43Z")
Hello all, does anyone have any experience they can share about doing automated testing of HPC clusters? As in, we would like to have a test suite that we can run on our cluster any time we make a configuration change or… --- ## [Installing Python Packages Using Jupyter Notebooks with Bridges OnDemand](https://ask.cyberinfrastructure.org/t/installing-python-packages-using-jupyter-notebooks-with-bridges-ondemand/1337)
**Author:** [@Adam\_Anderson](https://ask.cyberinfrastructure.org/u/Adam_Anderson)\ **Replies:** 3\ **Last updated:** [May 21, 2020, 6:43pm UTC](https://ask.cyberinfrastructure.org/t/installing-python-packages-using-jupyter-notebooks-with-bridges-ondemand/1337 "2020-05-21T18:43:04Z")
I’m trying to install pytorch and pytesseract on PSC Bridges and I want to use that on a compute node, running in Jupyter notebook via OnDemand. I’ve installed both packages using the login node and pip install --user py… --- ## [Which HPC applications to use for GPU benchmarking](https://ask.cyberinfrastructure.org/t/which-hpc-applications-to-use-for-gpu-benchmarking/1353)
**Author:** [@ktrn](https://ask.cyberinfrastructure.org/u/ktrn)\ **Replies:** 2\ **Last updated:** [May 13, 2020, 2:06am UTC](https://ask.cyberinfrastructure.org/t/which-hpc-applications-to-use-for-gpu-benchmarking/1353 "2020-05-13T02:06:38Z")
Almost every year a new type of GPU is released. Our research community is highly diverse and runs both single and double precision GPU applications. We would like to compare various kinds of GPUs and see how much a new … --- ## [Anyone else have failures with Intel 2019-mpi with Mellanox IB for # of cores \> 300?](https://ask.cyberinfrastructure.org/t/anyone-else-have-failures-with-intel-2019-mpi-with-mellanox-ib-for-of-cores-300/1035)
**Author:** [@ewalter](https://ask.cyberinfrastructure.org/u/ewalter)\ **Replies:** 8\ **Last updated:** [April 7, 2020, 4:43pm UTC](https://ask.cyberinfrastructure.org/t/anyone-else-have-failures-with-intel-2019-mpi-with-mellanox-ib-for-of-cores-300/1035 "2020-04-07T16:43:12Z")
Just curious if anyone has any comments regarding a problem we are seeing on our now skylake cluster. All MPI/compiler combinations including intel-2018, intel-2019, gcc-9.1.0 with openmpi/mvapich2 as well as intel-2018… [Next page](https://ask.cyberinfrastructure.org/tag/qow/175.md?match_all_tags=true&page=1&tags%5B%5D=qow)