Ask.Cyberinfrastructure

Is there a way to do startup and cleanup tasks with an SGE task array?

hpc-getting-started
parameter-sweep
scripting
scheduler
sge
#1

Is there a way to run one time commands at the beginning before the first task in the array or after the last task?

Doing things like creating a new directory before hand to put all the output files in, or being able to tar and move them to a long term storage location would be very helpful.

0 Likes

How can I manage a large number of very small independent jobs of varying length with some order dependencies?
#2

There are different, but combinable options for doing so by either when the first or last task starts
and for when all tasks finish:

When only concerned about when tasks start:
The key is that SGE array jobs set some additional variables
$SGE_TASK_FIRST, $SGE_TASK_LAST, $SGE_STEP_SIZE.

#!/bin/bash
#$ -t 1-100
if [ $SGE_TASK_ID -eq $SGE_TASK_FIRST ] ; then
# do first-task stuff here
fi

# do normal processing here

if [ $SGE_TASK_ID -q $SGE_TASK_LAST ]; then
# do last-task stuff here
fi

When concerned about when the last job finishes:
you can use the --sync flag in a script

qsub --sync y jobscript.sh >> exitstatus.log
# do things after

Will wait on the submitted job(s) to complete and report its exit code before closing.
With an array job it will wait on, and report on all of them.

0 Likes

How can I manage a large number of very small independent jobs of varying length with some order dependencies?
#3

This seems rather clunky are there better ways?

0 Likes