HPC Customization

NGSANE's main aim is to facilitate the submission of bioinformatics analysis tasks to a high-performance-compute-cluster (hpc). In its simplest form, the end user can execute bioinformatics analyses on very large datasets with a simple NGSANE command. The NGSANE framework is also highly customisable without the need to modify the underlying scripts, thus accommodating the needs of advanced users in an efficient manner.

This section describes how to customise NGSANE's execution on various hpc platforms.

Setup the system

NGSANE currently supports Grid Engine (SGE) and Portable Batch System (PBS). Which system is used must be defined in the conf/header.sh file (see Setup guide)

##############################################################
# System info
##############################################################
SUBMISSIONSYSTEM="PBS"                            # SGE or PBS
QUEUEWAIT=" -W depend=afterok:"                   # PBS
QUEUEWAITSEP=":"
#QUEUEWAIT=" -hold_jid "                          # SGE
#QUEUEWAITSEP=","
#QUEUEPARENV="smp"

If you are using SGE you also need to find out about the parallel environment setup on your system

-bash-4.1$ qconf -spl
orte

so you need to change

QUEUEPARENV="orte"

Customization

Customization is generally achieved by modifying the QSUBEXTRA parameter in the project's config file. Which is handed to qsub as $SADDITIONAL in jobSubmission.sh via prepareJobSubmission.sh.

PBS

	command="qsub $JOBIDS -V -j oe -o $SOUTPUT -w $(pwd) -l $SNODES -l vmem=$SMEMORY \
		-N $SNAME -l walltime=$SWALLTIME $TMPFILE $SADDITIONAL -l prologue=$SOUTPUT.sh	"

SGE

	command="qsub $HOLD_JID -V -S /bin/bash -j y -o $SOUTPUT -cwd -pe smp $SCPU -l h_vmem=$SMEMORY \
	    -N $SNAME -l h_rt=$SWALLTIME $SADDITIONAL $TMPFILE" 

This lets you easily define

• Different Queues QSUBEXTRA=" -q myspecialqueue"
• Wait-for parameters QSUBEXTRA=" -W depend=afterok:2182678:2182679" or QSUBEXTRA=" -hold_jid=2182678:2182679" more details
• Job Priority QSUBEXTRA= "-p priority"

Defining Job-dependency-queues

QSUBEXTRA also lets you define job dependencies, e.g. if you want to run bowtie followed by bigwig you can do this by adding the bowtie-job-ids to the config file for bigwig.

JOBIDSBOWTIE=$( trigger.sh configBOWTIE2.txt forcearmed | grep "Jobnumber" | cut -d " " -f 2 | tr '\n' ':' | sed 's/:$//g' )
echo 'QSUBEXTRA=$QSUBEXTRA'"\" $QUEUEWAIT${JOBIDSBOWTIE//:/$QUEUEWAITSEP}\"" >>configBIGWIG.txt
JOBIDSBW=$( trigger.sh configBIGWIG.txt forcearmed | grep "Jobnumber" | cut -d " " -f 2 | tr '\n' ':' | sed 's/:$//g' )

See smokebox files for more examples, e.g. testRUNCHIPSEQ.qsub. For a more permanent solution to define pipelines see trigger.sh's RUNTOPHATCUFFHTSEQ-section