# \#python **URL:** https://ask.cyberinfrastructure.org/tag/python/10.md [Latest](https://ask.cyberinfrastructure.org/latest.md) · [Categories](https://ask.cyberinfrastructure.org/categories.md) · [Tags](https://ask.cyberinfrastructure.org/tags.md) --- ## [Uv vs. conda vs. virtualenv](https://ask.cyberinfrastructure.org/t/uv-vs-conda-vs-virtualenv/4232)
**Author:** [@ktrn](https://ask.cyberinfrastructure.org/u/ktrn)\ **Replies:** 2\ **Last updated:** [August 4, 2025, 1:34pm UTC](https://ask.cyberinfrastructure.org/t/uv-vs-conda-vs-virtualenv/4232 "2025-08-04T13:34:43Z")
Dear colleagues, Have you used uv to manage your Python environment (https://github.com/astral-sh/uv)? It seems to work very fast, but I wonder if there are any challenges or downsides to using this package management s… --- ## [How to access files on cloud (gs, aws, etc) from code, such like Python, R?](https://ask.cyberinfrastructure.org/t/how-to-access-files-on-cloud-gs-aws-etc-from-code-such-like-python-r/4011)
**Author:** [@yshen16](https://ask.cyberinfrastructure.org/u/yshen16)\ **Replies:** 0\ **Last updated:** [August 21, 2024, 1:19pm UTC](https://ask.cyberinfrastructure.org/t/how-to-access-files-on-cloud-gs-aws-etc-from-code-such-like-python-r/4011 "2024-08-21T13:19:59Z")
Hi Folks, I’d like to ask if anyone would like to share some experience on how to get access to the file system on cloud, such as google cloud storage or aws S3 bucket, or azure, etc. directly from code, such as python,… --- ## [Tell me, why I should use Anaconda instead of Python Virtual Environment?](https://ask.cyberinfrastructure.org/t/tell-me-why-i-should-use-anaconda-instead-of-python-virtual-environment/4054)
**Author:** [@jfossot](https://ask.cyberinfrastructure.org/u/jfossot)\ **Replies:** 1\ **Last updated:** [September 17, 2024, 1:44pm UTC](https://ask.cyberinfrastructure.org/t/tell-me-why-i-should-use-anaconda-instead-of-python-virtual-environment/4054 "2024-09-17T13:44:49Z")
By my experience, setting up a conda environment is as complicated as it can get and I have to remember certain things to be able to activate and deactivate the environment. Also, the environment has to be turned on for … --- ## [Anaconda license situation at your site?](https://ask.cyberinfrastructure.org/t/anaconda-license-situation-at-your-site/4005)
**Author:** [@RonRahaman](https://ask.cyberinfrastructure.org/u/RonRahaman)\ **Replies:** 4\ **Last updated:** [September 4, 2024, 2:27pm UTC](https://ask.cyberinfrastructure.org/t/anaconda-license-situation-at-your-site/4005 "2024-09-04T14:27:08Z")
Hi all, Our site is trying to deal with Anaconda’s ongoing changes to its terms of service, which might make it prohibitively expensive to provide Anaconda as a software module on our clusters. Have any of your sites d… --- ## [Resources for GPU and Google JAX programming](https://ask.cyberinfrastructure.org/t/resources-for-gpu-and-google-jax-programming/3133)
**Author:** [@sam\_dt](https://ask.cyberinfrastructure.org/u/sam_dt)\ **Replies:** 0\ **Last updated:** [May 17, 2024, 2:27pm UTC](https://ask.cyberinfrastructure.org/t/resources-for-gpu-and-google-jax-programming/3133 "2024-05-17T14:27:23Z")
When I first started my project in Google JAX, I knew that that Google JAX was a pretty new coding library made for GPUs and that examples and teaching materials would be scarce for that. And in addition, I was pretty ne… --- ## [Examine specific lines of code for memory profiling](https://ask.cyberinfrastructure.org/t/examine-specific-lines-of-code-for-memory-profiling/2963)
**Author:** [@shrek](https://ask.cyberinfrastructure.org/u/shrek)\ **Replies:** 1\ **Last updated:** [September 30, 2023, 7:50pm UTC](https://ask.cyberinfrastructure.org/t/examine-specific-lines-of-code-for-memory-profiling/2963 "2023-09-30T19:50:44Z")
How can I obtain detailed insights for analyzing hardware memory usage in fine detail within a simple Python function as shown below, especially when examining specific lines of code contributing to memory growth, simila… --- ## [Coldfusion - Python WebApp networking question](https://ask.cyberinfrastructure.org/t/coldfusion-python-webapp-networking-question/2950)
**Author:** [@chris\_blanton1](https://ask.cyberinfrastructure.org/u/chris_blanton1)\ **Replies:** 0\ **Last updated:** [September 22, 2023, 4:08pm UTC](https://ask.cyberinfrastructure.org/t/coldfusion-python-webapp-networking-question/2950 "2023-09-22T16:08:38Z")
We had a question come in for consultation and I’m not sure where to start so I’m asking for thoughts and resources if anyone can share: What is the best and most secure way to let ColdFusion web application to listen … --- ## [Profile Python processes across multiple cores](https://ask.cyberinfrastructure.org/t/profile-python-processes-across-multiple-cores/2867)
**Author:** [@cec5550](https://ask.cyberinfrastructure.org/u/cec5550)\ **Replies:** 2\ **Last updated:** [August 17, 2023, 3:30pm UTC](https://ask.cyberinfrastructure.org/t/profile-python-processes-across-multiple-cores/2867 "2023-08-17T15:30:38Z")
I am running many different processes with Python’s multiprocessing module. What would be the best way to profile each process so that I could determine the limiting process for my Python program? I would like to see the… --- ## [Create a SQLite database on TACC](https://ask.cyberinfrastructure.org/t/create-a-sqlite-database-on-tacc/2905)
**Author:** [@sohil.shrestha](https://ask.cyberinfrastructure.org/u/sohil.shrestha)\ **Replies:** 1\ **Last updated:** [August 16, 2023, 4:28pm UTC](https://ask.cyberinfrastructure.org/t/create-a-sqlite-database-on-tacc/2905 "2023-08-16T16:28:56Z")
I am trying to train a deep learning model in TACC. My project involves creating a sqlite3 database . I get the following error when i try to create a database: (sqlite3.OperationalError) disk I/O error \[SQL: PRAGMA ta… --- ## [Alternative to Posit Team](https://ask.cyberinfrastructure.org/t/alternative-to-posit-team/2872)
**Author:** [@ERWIN\_LARES](https://ask.cyberinfrastructure.org/u/ERWIN_LARES)\ **Replies:** 1\ **Last updated:** [July 16, 2023, 1:16pm UTC](https://ask.cyberinfrastructure.org/t/alternative-to-posit-team/2872 "2023-07-16T13:16:47Z")
hello everyone, I was wondering if anybody has researched alternatives to Posit Workbench and Connect products. --- ## [How to use multiple GPU node to train/fine-tune large language models?](https://ask.cyberinfrastructure.org/t/how-to-use-multiple-gpu-node-to-train-fine-tune-large-language-models/2816)
**Author:** [@sohil.shrestha](https://ask.cyberinfrastructure.org/u/sohil.shrestha)\ **Replies:** 3\ **Last updated:** [May 31, 2023, 7:00pm UTC](https://ask.cyberinfrastructure.org/t/how-to-use-multiple-gpu-node-to-train-fine-tune-large-language-models/2816 "2023-05-31T19:00:57Z")
I am using UT’s Texas Advanced Computing Center (TACC)'s resources to fine tune large language models. However, I am able to fine-tune it with 1-batch size making the fine-tuning extremely slow and unstable. My understa… --- ## [What qualifies as a virtual environment, and when is it appropriate to create a virtual environment?](https://ask.cyberinfrastructure.org/t/what-qualifies-as-a-virtual-environment-and-when-is-it-appropriate-to-create-a-virtual-environment/2246)
**Author:** [@toreliza](https://ask.cyberinfrastructure.org/u/toreliza)\ **Replies:** 4\ **Last updated:** [May 12, 2023, 2:48pm UTC](https://ask.cyberinfrastructure.org/t/what-qualifies-as-a-virtual-environment-and-when-is-it-appropriate-to-create-a-virtual-environment/2246 "2023-05-12T14:48:08Z")
Requests to set up virtual environments (python, containers, os-related, for example) seem to be increasing in frequency. A workshop in which I recently participated required attendees to create a virtual environment fo… --- ## [Creating a Python Virtual Environment](https://ask.cyberinfrastructure.org/t/creating-a-python-virtual-environment/2063)
**Author:** [@Christopher\_Nadeau](https://ask.cyberinfrastructure.org/u/Christopher_Nadeau)\ **Replies:** 2\ **Last updated:** [May 11, 2023, 2:01pm UTC](https://ask.cyberinfrastructure.org/t/creating-a-python-virtual-environment/2063 "2023-05-11T14:01:31Z")
What is a Virtual Environment? A virtual environment is a way to isolate a specific working environment from the rest of your system. It is highly suggested to use a Python virtual environment when working on computing p… --- ## [Rutgers University is hiring a Director of Research Support!](https://ask.cyberinfrastructure.org/t/rutgers-university-is-hiring-a-director-of-research-support/2713)
**Author:** [@JanaeBaker](https://ask.cyberinfrastructure.org/u/JanaeBaker)\ **Replies:** 0\ **Last updated:** [March 20, 2023, 10:06pm UTC](https://ask.cyberinfrastructure.org/t/rutgers-university-is-hiring-a-director-of-research-support/2713 "2023-03-20T22:06:36Z")
The Rutgers’ Office of Advanced Research Computing (OARC) is on the look for a Director of Research Support to join our growing team! Rutgers is a leading national research university and the State of New Jersey’s preem… --- ## [Systems Administrators in Pittsburgh: Servers, Clusters and Supercomputers for Computational Biochemistry](https://ask.cyberinfrastructure.org/t/systems-administrators-in-pittsburgh-servers-clusters-and-supercomputers-for-computational-biochemistry/2605)
**Author:** [@DESRESCareers](https://ask.cyberinfrastructure.org/u/DESRESCareers)\ **Replies:** 0\ **Last updated:** [November 7, 2022, 6:12am UTC](https://ask.cyberinfrastructure.org/t/systems-administrators-in-pittsburgh-servers-clusters-and-supercomputers-for-computational-biochemistry/2605 "2022-11-07T06:12:15Z")
Exceptional sysadmins sought to manage systems infrastructure, networks, and data center operations at a state-of-the-art facility in Pittsburgh, PA. Successful hires will be responsible for bring-up and ongoing mainten… --- ## [Systems Administrators in New York City: Servers, Clusters and Supercomputers for Computational Biochemistry](https://ask.cyberinfrastructure.org/t/systems-administrators-in-new-york-city-servers-clusters-and-supercomputers-for-computational-biochemistry/2350)
**Author:** [@DESRESCareers](https://ask.cyberinfrastructure.org/u/DESRESCareers)\ **Replies:** 0\ **Last updated:** [May 18, 2022, 11:42am UTC](https://ask.cyberinfrastructure.org/t/systems-administrators-in-new-york-city-servers-clusters-and-supercomputers-for-computational-biochemistry/2350 "2022-05-18T11:42:31Z")
Exceptional sysadmins sought to manage systems infrastructure, networks, and information security for a New York–based interdisciplinary research group. Successful hires will be responsible for our servers and cluster e… --- ## [What is the difference between the subprocess module and the os module in python? (For running shell commands)](https://ask.cyberinfrastructure.org/t/what-is-the-difference-between-the-subprocess-module-and-the-os-module-in-python-for-running-shell-commands/2261)
**Author:** [@tolu\_058](https://ask.cyberinfrastructure.org/u/tolu_058)\ **Replies:** 1\ **Last updated:** [March 22, 2022, 8:20pm UTC](https://ask.cyberinfrastructure.org/t/what-is-the-difference-between-the-subprocess-module-and-the-os-module-in-python-for-running-shell-commands/2261 "2022-03-22T20:20:22Z")
I want to incorporate shell commands in my python HPC script and I was wondering which python module is better to use - os or subprocess? os seems fairly easier to use but subprocess is the recommended module. --- ## [Using anaconda.org to find software](https://ask.cyberinfrastructure.org/t/using-anaconda-org-to-find-software/1746)
**Author:** [@william.wilson](https://ask.cyberinfrastructure.org/u/william.wilson)\ **Replies:** 0\ **Last updated:** [February 18, 2021, 5:10pm UTC](https://ask.cyberinfrastructure.org/t/using-anaconda-org-to-find-software/1746 "2021-02-18T17:10:21Z")
Here at NAU we leverage Anaconda to assist researchers with setting up environments for python. For more information on using Anaconda at NAU, see the following article - https://ask.cyberinfrastructure.org/t/how-do-i-us… --- ## [What resources are used in academia to publish apps](https://ask.cyberinfrastructure.org/t/what-resources-are-used-in-academia-to-publish-apps/1831)
**Author:** [@ktrn](https://ask.cyberinfrastructure.org/u/ktrn)\ **Replies:** 1\ **Last updated:** [April 12, 2021, 8:46pm UTC](https://ask.cyberinfrastructure.org/t/what-resources-are-used-in-academia-to-publish-apps/1831 "2021-04-12T20:46:20Z")
We have more and more researchers who would like to publish an app (to be used worldwide), that requires a significant amount of memory, computational resources (multiple cores, GPU, etc. ). Some examples include R Shiny… --- ## [How do I use scikit-learn on the Princeton HPC Clusters?](https://ask.cyberinfrastructure.org/t/how-do-i-use-scikit-learn-on-the-princeton-hpc-clusters/1766)
**Author:** [@rrkknight3](https://ask.cyberinfrastructure.org/u/rrkknight3)\ **Replies:** 1\ **Last updated:** [March 1, 2021, 12:24pm UTC](https://ask.cyberinfrastructure.org/t/how-do-i-use-scikit-learn-on-the-princeton-hpc-clusters/1766 "2021-03-01T12:24:17Z")
I’m getting started with machine learning and have been told that scikit-learn is a good place to start. How do I use this package on Princeton’s HPC clusters? --- ## [Installing cupy](https://ask.cyberinfrastructure.org/t/installing-cupy/1482)
**Author:** [@antoinearnoud](https://ask.cyberinfrastructure.org/u/antoinearnoud)\ **Replies:** 1\ **Last updated:** [August 26, 2020, 1:44pm UTC](https://ask.cyberinfrastructure.org/t/installing-cupy/1482 "2020-08-26T13:44:56Z")
Hi, I am using the python module OPT. I would like to use the GPU implementation of some functions. It seems it requires the module cupy. I tried to install cupy (pip install cupy) but I get an error: ERROR: CUDA coul… --- ## [Installing Python Packages Using Jupyter Notebooks with Bridges OnDemand](https://ask.cyberinfrastructure.org/t/installing-python-packages-using-jupyter-notebooks-with-bridges-ondemand/1337)
**Author:** [@Adam\_Anderson](https://ask.cyberinfrastructure.org/u/Adam_Anderson)\ **Replies:** 3\ **Last updated:** [May 21, 2020, 6:43pm UTC](https://ask.cyberinfrastructure.org/t/installing-python-packages-using-jupyter-notebooks-with-bridges-ondemand/1337 "2020-05-21T18:43:04Z")
I’m trying to install pytorch and pytesseract on PSC Bridges and I want to use that on a compute node, running in Jupyter notebook via OnDemand. I’ve installed both packages using the login node and pip install --user py… --- ## [Porting a GNU Parallel parallelization onto a Slurm cluster](https://ask.cyberinfrastructure.org/t/porting-a-gnu-parallel-parallelization-onto-a-slurm-cluster/1214)
**Author:** [@StrongPiccolo](https://ask.cyberinfrastructure.org/u/StrongPiccolo)\ **Replies:** 2\ **Last updated:** [January 27, 2020, 11:07pm UTC](https://ask.cyberinfrastructure.org/t/porting-a-gnu-parallel-parallelization-onto-a-slurm-cluster/1214 "2020-01-27T23:07:22Z")
I’m running a Sage MATH script I wrote with GNU parallel to get local parallelization, however I am currently looking to scale up to a cluster that uses Slurm as a workload manager. I am using a simple GNU parallel scri… --- ## [How do I get started with Python on HPC?](https://ask.cyberinfrastructure.org/t/how-do-i-get-started-with-python-on-hpc/871)
**Author:** [@vsoch](https://ask.cyberinfrastructure.org/u/vsoch)\ **Replies:** 5\ **Last updated:** [November 21, 2019, 12:01am UTC](https://ask.cyberinfrastructure.org/t/how-do-i-get-started-with-python-on-hpc/871 "2019-11-21T00:01:18Z")
If I am new to HPC (but perhaps have used Python) what modules should I take a look at? --- ## [How do I run mpi4py?](https://ask.cyberinfrastructure.org/t/how-do-i-run-mpi4py/943)
**Author:** [@vsoch](https://ask.cyberinfrastructure.org/u/vsoch)\ **Replies:** 4\ **Last updated:** [June 21, 2019, 3:44pm UTC](https://ask.cyberinfrastructure.org/t/how-do-i-run-mpi4py/943 "2019-06-21T15:44:53Z")
I’m having problems running mpi4py, can anyone help with a basic example? Thank you! --- ## [Using LMOD to load Python libraries from a user's home directory](https://ask.cyberinfrastructure.org/t/using-lmod-to-load-python-libraries-from-a-users-home-directory/108)
**Author:** [@gwcunhcie](https://ask.cyberinfrastructure.org/u/gwcunhcie)\ **Replies:** 3\ **Last updated:** [November 26, 2018, 8:26am UTC](https://ask.cyberinfrastructure.org/t/using-lmod-to-load-python-libraries-from-a-users-home-directory/108 "2018-11-26T08:26:56Z")
I’ve installed a local Python3 library at /home/$USER/python3lib. How can I use LMOD to load this library? --- ## [How do I install Python 2.7 on my Comet allocation and where can I store data long term?](https://ask.cyberinfrastructure.org/t/how-do-i-install-python-2-7-on-my-comet-allocation-and-where-can-i-store-data-long-term/599)
**Author:** [@cydburrows](https://ask.cyberinfrastructure.org/u/cydburrows)\ **Replies:** 2\ **Last updated:** [November 16, 2018, 9:20pm UTC](https://ask.cyberinfrastructure.org/t/how-do-i-install-python-2-7-on-my-comet-allocation-and-where-can-i-store-data-long-term/599 "2018-11-16T21:20:59Z")
A new XSEDE user asks: Is there a way to install Python 2.7 and specific Python packages to the Comet computing environment, and to analyze data using this software? Also, what is the best place to store data long-term … --- ## [What is the most efficient Python package to run a parallel job over multiple nodes?](https://ask.cyberinfrastructure.org/t/what-is-the-most-efficient-python-package-to-run-a-parallel-job-over-multiple-nodes/171)
**Author:** [@ktrn](https://ask.cyberinfrastructure.org/u/ktrn)\ **Replies:** 3\ **Last updated:** [August 20, 2018, 9:44pm UTC](https://ask.cyberinfrastructure.org/t/what-is-the-most-efficient-python-package-to-run-a-parallel-job-over-multiple-nodes/171 "2018-08-20T21:44:38Z")
I would like to parallelize my Python script over multiple nodes on the cluster. Which Python package is most efficient for this purpose? CURATOR: Katia --- ## [How do I know if my job is parallelizing when submitted as a batch script?](https://ask.cyberinfrastructure.org/t/how-do-i-know-if-my-job-is-parallelizing-when-submitted-as-a-batch-script/222)
**Author:** [@ktrn](https://ask.cyberinfrastructure.org/u/ktrn)\ **Replies:** 3\ **Last updated:** [June 8, 2018, 12:41pm UTC](https://ask.cyberinfrastructure.org/t/how-do-i-know-if-my-job-is-parallelizing-when-submitted-as-a-batch-script/222 "2018-06-08T12:41:48Z")
I am trying to parallelize my Python job, but I am not sure if it runs using many cores. I looked at the time of the execution and the time of the execution is a little shorter than when I submit the same job without pa…