Ask.Cyberinfrastructure

How do I estimate wall clock time?

scheduler

#1

Many resource schedulers provide a way to indicate the duration of wall clock time a job will require. What are tips and tricks for estimating the duration to specify prior to execution of a job?

CURATOR: Scott Yockel


#2

ANSWER: It would depend on application. But there are common approaches:
If there is a loop (for example with iterator ranging from 1 to a number of simulations), one can run a job for a few iterations (assuming that all iterations take approximately the same time) and then based on this estimate the time it would take to run the full job.

In some other cases running a program for a smaller input allows estimating the time for a larger input.


#3

In addition to the situations described by Katia, one might need to estimate how a job might scale. There are several options, depending on what data, if any, is available. One can extrapolate from as little as two timing instances, say completion on two nodes and completion on 8 nodes (or core, or other resource ‘unit’). If one knows how long a serial run takes, and has data points of runtimes from a completely different computation, those three numbers can be used to estimate parallel performance of the serial run (assuming likeness of parallelizability between the computations). If the reason a wall clock time is needed is for allocation purposes on a ‘new’ system, one might report in scaling percentages based on runs made on another cluster.