The queueing system on grommet is designed to maximize machine job throughput while keeping the load at a reasonable level. It tries to balance the usage among the different groups on grommet to help ensure that everyone's job gets a chance to run. The system is not perfect. Work is ongoing to improve the "fairness" of the job scheduler.

Details of the queueing system setup and instructions on running jobs in the queue can be found below.

If you have problems getting your script to work, let me know (pmcmahon"at"udel.edu) and we'll figure out what's going wrong.

This document contains sample qsub scripts. I hope to gather sample scripts for each of the major packages we run on grommet (MSI software, gaussian, jaguar, molpro, amber, etc.). If you have a working script to contribute, let me know.

There also links to additional documentation (beyond the man pages) and descriptions of some of the common (and possibly useful) qsub options.

This page is a work in progress. If there are any glaring omissions on this page, please let me know and I will try to fill in the gaps.

Queues and Job Scheduling on Grommet

Currently there are three queues on grommet:

The queueing system attempts to keep the average background load on grommet to around 12 (1.5 x the number of processors in the system). The total number of cpus assigned to jobs running in the main and fourhour queues is limited to 12. Jobs in the debug queue are not subject to this processor restriction.

Running jobs on grommet with PBS

1. Use your favorite editor (vi, emacs, nedit, jot) to create a shell script (text file) containing the commands needed to run your job. For many programs these will be the same commands you would use to run it at the command line. For some programs, the commands may be slightly different. Some experimentation may be necessary to work it out.

2. Submit the job to the default queue using the qsub command:

qsub myjob

where myjob is the name of the script you created to run your job.

3. qsub will return the jobid of your job if it is successfully queued. The jobid will have the format:
   
   ##.grommet.chem.udel.edu
   
   where ## is a non-negative integer. You can use either the full jobid or just the initial ## to refer to your job in any PBS command that takes a jobid as input (qstat, qdel, qhold, qrls).

4. Use the qstat command to monitor the progress of your job.
   
   qstat -a
   
   shows a brief listing of all the jobs in the queue.
   
   qstat -f jobid
   
   shows more detailed information about the job specified.

Command Summary

optional arguments are enclosed in [ ].

• qsub: submit a job to the queue for execution. Queues on grommet are main (default) and debug (not active). If no, queue is specified, jobs are submitted into the default queue (main).
  
  qsub [-q queuename] myscript
• qdel: delete a job from the queue.
  
  qdel job_id
• qstat: view status of jobs/queues/server.
  
  job status (use -a for an alternate listing, -f for more details):
  
  qstat [-f] [job_id]
  qstat -a
  queue status (use -Qf for more details, -q for alternate listing):
  
  qstat -Q[f] [queuename]
  qstat -q [queuename]
  server status (use -Bf for more details):
  
  qstat -B[f]

Useful qsub options

Anatomy of a PBS Script

Sample Scripts

Gamess: This script runs a sequential gamess job from the input file myjob.inp which is in the directory from which the qsub command was issued. The npri -w command preceeding the seqgms command causes the job to be run "weightless" in the background. The job will be checkpointed every 2 hours (-c c) so it can be restarted if the system goes down unexpectedly, and the standard output and error of the script will be returned in the jobname.o## file (-j oe).

#!/bin/tcsh -f
#PBS -c c -j oe
cd $PBS_O_WORKDIR
npri -w seqgms myjob

Gaussian 98: this script will run a 2 processor job (-l ncpus=2), checkpoint the job every 2 hours (-c c) and merge the standard error and standard output of the script into one file jobname.o## (-j oe). The Gaussian input file is myjob.com in the direcory where qsub was called (i.e. where the script is) and the output from g98 (and standard error from g98) is directed to myjob.out. For a single processor job, leave out the -l ncpus=2, or change the number for a job requiring more than 2 processors.

This script works for csh and tcsh users:

#!/bin/tcsh -f
#PBS -c c -j oe -l ncpus=2
cd $PBS_O_WORKDIR
setg98
npri -w g98 < myjob.com >& myjob.out

This script should work for ksh users:

#!/bin/ksh -f
#PBS -c c -j oe -l ncpus=2
cd $PBS_O_WORKDIR
g98root=/usr/programs
. $g98root/g98/bsd/g98.profile
npri -w g98 < myjob.com >& myjob.out

This script (for csh and tcsh users) shows the use of the short form of the gaussian submission command. The short form of the command can also be used in a script formatted for ksh. This form assumes that your input file is called myjob.com and will create a gaussian output file called myjob.log.:

#!/bin/tcsh -f
#PBS -c c -j oe -l ncpus=2
cd $PBS_O_WORKDIR
setg98
npri -w g98 myjob.com

Jaguar: This script runs a jaguar job from the jaguar input file myjob.in which is in the directory from which the qsub command was issued. The -w command on the jaguar run command causes jaguar to execute the job in the foreground rather than in the background. Schrodinger (makers of Jaguar) claim that the -w command will have the same effect on all jaguar commands (jaguar run, jaguar batch, etc.). Again the job will be checkpointed every 2 hours (-c c) so it can be restarted if the system goes down unexpectedly, and the standard output and error of the script will be returned in the jobname.o## file (-j oe).

#!/bin/tcsh -f
#PBS -c c -j oe
cd $PBS_O_WORKDIR
npri -w jaguar run -w myjob

All About Grommet...	Still to come...
Grommet Home Page Participating Faculty Funding Sources Research from Grommet Description of the Facility Software Available on Grommet Online Help for Grommet Users	Interesting Links for Grommet Users and others

This page is maintained by Patrick McMahon.
Last Updated: 12 July 2008
The URL for this page is: http://www.udel.edu/chem/grommet/help/queues.html