TORQUE: Review configuration

[vlj91]

The default TORQUE configuration we have seems to not do scheduling entirely correctly, for example in testing it is possible to:

• submit 16 x 2-core jobs, that will then run on a single 16-core compute node

The autoscaler also assumes that multiple jobs cannot be run on a single compute host (this is usually the case for TORQUE) - so the autoscaler is calculating the number of nodes required based on a single job per node, when in fact the scheduler is allowing us currently to run multiple jobs on a single compute host.

There is probably some series of configuration that allocates jobs appropriately

Possibly useful (thanks @mjtko) - http://www.supercluster.org/pipermail/torqueusers/2012-May/014636.html

[vlj91]

Got the following configuration which sets up multiple queues and distributes jobs to the correct queue:

## apply queue defaults
qmgr -c 'set server queue_centric_limits=True'

## create the routing queue
qmgr -c 'create queue default'
qmgr -c 'set queue default queue_type=Route'
qmgr -c 'set queue default route_destinations=serial'
qmgr -c 'set queue default route_destinations+=smp'
qmgr -c 'set queue default route_destinations+=mpi'
qmgr -c 'set queue default enabled=True'
qmgr -c 'set queue default started=True'

## create the serial queue
qmgr -c 'create queue serial'
qmgr -c 'set queue serial queue_type=Execution'
qmgr -c 'set queue serial Priority=100'
qmgr -c 'set queue serial resources_max.nodes=1'
qmgr -c 'set queue serial resources_max.ncpus=1'
qmgr -c 'set queue serial resources_max.procct=1'
qmgr -c 'set queue serial resources_default.nodes=1'
qmgr -c 'set queue serial resources_default.ncpus=1'
qmgr -c 'set queue serial resources_default.procct=1'
qmgr -c 'set queue serial enabled=True'
qmgr -c 'set queue serial started=True'

## create the smp queue
qmgr -c 'create queue smp'
qmgr -c 'set queue smp queue_type=Execution'
qmgr -c 'set queue smp Priority=95'
qmgr -c 'set queue smp resources_max.nodes=1'
qmgr -c 'set queue smp resources_default.nodes=1'
qmgr -c 'set queue smp enabled=True'
qmgr -c 'set queue smp started=True'

## create the mpi queue
qmgr -c 'create queue mpi'
qmgr -c 'set queue mpi queue_type=Execution'
qmgr -c 'set queue mpi Priority=90'
qmgr -c 'set queue mpi resources_default.nodes=2'
qmgr -c 'set queue mpi resources_default.ncpus=2'
qmgr -c 'set queue mpi resources_default.procct=2'
qmgr -c 'set queue mpi resources_min.ncpus=2'
qmgr -c 'set queue mpi resources_min.procct=2'
qmgr -c 'set queue mpi enabled=True'
qmgr -c 'set queue mpi started=True'

[alces@login1(torquey-mctorqueface) ~]$ sleep 1 | qsub -l ncpus=2
17.login1.torquey-mctorqueface.prv.alces.network
[alces@login1(torquey-mctorqueface) ~]$ qstat
Job ID                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
17.login1                  STDIN            alces           00:00:00 C smp
[alces@login1(torquey-mctorqueface) ~]$ sleep 1 | qsub
18.login1.torquey-mctorqueface.prv.alces.network
[alces@login1(torquey-mctorqueface) ~]$ sleep 1 | qsub -l nodes=2:ppn=1
19.login1.torquey-mctorqueface.prv.alces.network
[alces@login1(torquey-mctorqueface) ~]$ qstat
Job ID                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
17.login1                  STDIN            alces           00:00:00 C smp
18.login1                  STDIN            alces           00:00:00 C serial
19.login1                  STDIN            alces           00:00:00 C mpi

Having trouble getting Torque to not allocate more job resources than an entire node can take (i.e 4 cores worth of jobs for a 2 core node).

[vlj91]

With the above configuration, no more jobs than the nodes resources allow for will be started on a single node - below shows a cluster with 4-core compute hosts, showing 4 single-core jobs running on 1 host and the others moving onto the next available host.

[alces@login1(testy-mctestface) ~]$ qstat -n1

login1.testy-mctestface.prv.alces.network:
                                                                                  Req'd       Req'd       Elap
Job ID                  Username    Queue    Jobname          SessID  NDS   TSK   Memory      Time    S   Time
----------------------- ----------- -------- ---------------- ------ ----- ------ --------- --------- - ---------
0.login1.testy-mctestf  alces       batch    sleep_240.sh       6454     1      1       --   01:00:00 C       --    flight-179
1.login1.testy-mctestf  alces       batch    sleep_240.sh       6535     1      1       --   01:00:00 C       --    flight-179
2.login1.testy-mctestf  alces       batch    sleep_240.sh       6425     2      2       --   01:00:00 C       --    flight-058+flight-179
3.login1.testy-mctestf  alces       serial   sleep_240.sh       6640     1      1       --        --  R  00:00:00   flight-179
4.login1.testy-mctestf  alces       serial   sleep_240.sh       6711     1      1       --        --  R  00:00:00   flight-179
5.login1.testy-mctestf  alces       serial   sleep_240.sh       6782     1      1       --        --  R  00:00:00   flight-179
6.login1.testy-mctestf  alces       serial   sleep_240.sh       6853     1      1       --        --  R  00:00:00   flight-179
7.login1.testy-mctestf  alces       serial   sleep_240.sh       6525     1      1       --        --  R  00:00:00   flight-058
8.login1.testy-mctestf  alces       serial   sleep_240.sh       6596     1      1       --        --  R  00:00:00   flight-058