How do I estimate if the hard time limit will be exceeded before submitting a job?



What’s a good way to find out ahead of time that my program might overrun the scheduler time limit, so I can plan to save state periodically?

CURATOR: Katia Oleinik



If the program contains a large loop, then running this loop just for a few iterations might provide you with a rough idea of how much time is spent for each iteration. Or, if the input dataset/system can be reduced to a smaller subset, then the wallclock time for this small subset can provide some clue about how much time might be needed for the whole dataset. When adding checkpoints to your program (to save the state) make sure you space them in a reasonable way, i.e. do not create a checkpoint too often - it will slow done the overall performance of your program.