# HPC job managers and migrating to the cloud **URL:** \ **Category:** Cloud Computing\ **Tags:** systems, cloud, aws, gce, azure, scheduler, researcher\ **Created:** [March 16, 2018, 1:19am UTC](https://ask.cyberinfrastructure.org/t/hpc-job-managers-and-migrating-to-the-cloud/91 "2018-03-16T01:19:04Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![raminder](https://ask.cyberinfrastructure.org/letter_avatar_proxy/v4/letter/r/58956e/32.png) [@raminder](https://ask.cyberinfrastructure.org/u/raminder)\ **Post date:** [March 16, 2018, 1:19am UTC](https://ask.cyberinfrastructure.org/t/hpc-job-managers-and-migrating-to-the-cloud/91/1 "2018-03-16T01:19:04Z")
What job managers are available, how do I easily move between them, and how do they map to tools in the cloud? I am currently using PBS to run my analysis and want to migrate to the cloud to collaborate for a multi-institution project. Is there any recommendation on the type of cloud service? And what tools I will need to learn to help with this migration. **CURATOR:** Raminder Singh ---
**Author:** ![aculich](https://ask.cyberinfrastructure.org/user_avatar/ask.cyberinfrastructure.org/aculich/32/79_2.png) [@aculich](https://ask.cyberinfrastructure.org/u/aculich)\ **Post date:** [July 17, 2018, 3:44pm UTC](https://ask.cyberinfrastructure.org/t/hpc-job-managers-and-migrating-to-the-cloud/91/2 "2018-07-17T15:44:35Z")
When considering whether and how to migrate your workload from an HPC cluster environment to the cloud you may first wish to check to see if your campus research computing group provides consulting to help with that. There are also resources beyond the campus at the national scale that may help such as the XSEDE [Extended Collaborative Support Service (ECSS)](https://www.xsede.org/for-users/ecss) and the [Software Community Gateways Institute (SGCI)](https://sciencegateways.org/). Those services are both available to help you determine what’s needed for a multi-institution project, especially if you have a workflow suitable as a Science Gateway that provides a web-frontend to HPC cluster resources (so that you don’t have to change your workflow, except to maybe switch to a different job scheduler such as SLURM available on the newer XSEDE clusters), or whether your workflow is better suited to a cloud environment that may not require a batch scheduler at all, but may require you to change your workflows and related software. ---
**Author:** ![jpessin1](https://ask.cyberinfrastructure.org/user_avatar/ask.cyberinfrastructure.org/jpessin1/32/339_2.png) [@jpessin1](https://ask.cyberinfrastructure.org/u/jpessin1)\ **Post date:** [July 20, 2018, 4:13pm UTC](https://ask.cyberinfrastructure.org/t/hpc-job-managers-and-migrating-to-the-cloud/91/3 "2018-07-20T16:13:59Z")
I’d recommend looking into intra/extra institutional support First (such as in @aculich’s great answer for an example). That said, There is nothing specific to the Cloud that will keep you from rolling your own setup using what ever you want. Additionally, pretty much all the cloud vendors also support, provide or encourage tools for easy setup and config with common schedulers. As well as alternatives based on server-less and/or container models, and pre-built ‘marketplace’ options _The tools according to the big three_: cnfCluster [CfnCluster — CfnCluster 1.6.1](https://cfncluster.readthedocs.io/en/latest/) AWS supported elastiCompute [http://gc3-uzh-ch.github.io/elasticluster/](http://gc3-uzh-ch.github.io/elasticluster/) GCE recommended (can also be used with AWS & private OpenStacks) [https://github.com/Azure/azure-quickstart-templates/tree/master/create-hpc-cluster-linux-cn](https://github.com/Azure/azure-quickstart-templates/tree/master/create-hpc-cluster-linux-cn) Azure all of which officially support SGE, Slurm, and Torque (PBSpro is an active request in both elastiCompute and cnfCluster) note: I’ve personally only dealt with AWS & GCE, mostly with SGE so YMMV. ---
**Author:** ![chrishill-regular](https://ask.cyberinfrastructure.org/letter_avatar_proxy/v4/letter/c/9e8a1a/32.png) [@chrishill-regular](https://ask.cyberinfrastructure.org/u/chrishill-regular)\ **Post date:** [July 23, 2018, 1:15am UTC](https://ask.cyberinfrastructure.org/t/hpc-job-managers-and-migrating-to-the-cloud/91/4 "2018-07-23T01:15:05Z")
A couple of quick comments and then an answer **Comments** 1. It is hard to tell what question is being asked from this post. It might be worth rewording to make it easier to get a focussed answer. The most useful answer may depend on what sort of workflow and what application/problem you are trying to solve. If its a specific question about spinning up a virtual cluster with a specific scheduler and software stack then the question should clarify that. 2. @raminder I think the post title might be better reading 3. The explanatory sentence 4. Overall it might be useful to clarify whether this is a narrow specific question such is “Is there a recipe I can use to create a PBS cluster on AWS?” or whether the question is broader like “What cloud platforms can I use for a PBS workflow that I currently execute on a bare metal cluster?”. **Now an answer** Assuming you are asking about using cloud resources with traditional HPC style workload managers then the answer is that nearly all cloud platforms in their most basic form provide networked virtual machines. As such you can configure pretty much any scheduler software that you can use on networked physical machines. As noted above > [@jpessin1](#): > > _The tools according to the big three_ : > cnfCluster [CfnCluster — CfnCluster 1.6.1](https://cfncluster.readthedocs.io/en/latest/) > AWS supported > > elastiCompute [http://gc3-uzh-ch.github.io/elasticluster/](http://gc3-uzh-ch.github.io/elasticluster/) > GCE recommended (can also be used with AWS & private OpenStacks) each of AWS, GCE and Azure provide some tools to streamline the process of launching multiple virtual machines in a manner that they can form a cluster. There are also a number of independent tools that can perform similar tasks. Some of these include - [StarCluster](http://star.mit.edu/cluster/), an early effort but still widely used. - [Terraform](https://www.terraform.io), a more general tool You can also script the steps of creating a cluster with some work yourself. If you are more looking for a service then there are several activities trying to address this. For example - [Rescale](https://www.rescale.com) provide a commercial service that utilizes cloud resources to streamline running many typical research computing application (e.g. [Gaussian](http://gaussian.com), [NAMD](http://www.ks.uiuc.edu/Research/namd/) etc…) workflows that are normally executed through job managers like PBS. ---
**Author:** ![boyd](https://ask.cyberinfrastructure.org/letter_avatar_proxy/v4/letter/b/dfb087/32.png) [@boyd](https://ask.cyberinfrastructure.org/u/boyd)\ **Post date:** [September 3, 2018, 3:04am UTC](https://ask.cyberinfrastructure.org/t/hpc-job-managers-and-migrating-to-the-cloud/91/5 "2018-09-03T03:04:06Z")
Another couple of options to mention are alces flight (AWS & Azure) and CloudyCluster (AWS and soon GCP, autoscaling with the CCQ meta-scheduler).