Magnesium

Our little cluster

Introduction

• 2.4 GHz AMD64 CPU's ("3400+") on Asus K8V SE Deluxe K8T800 M/B's
• 1.5 GB DDR333 RAM (11 nodes) / 3.0 GB DDR333 RAM (4 nodes)
• 340 GB /wrk, RAID-0, XFS (2 × Seagate Barracuda 7200.7 SATA)

The nodes are connected by pseudo-gigabit on-board ethernet and a ZyXEL GS-1116 switch. The latency is 18 µs (not that bad), the maximum throughput 460 Mbps (not that good) for packet sizes over 1 MB (TCP, see figure, tested with NetPIPE).

See below for latest news and changes.

Index of this page

• Introduction
• Index
• Very brief rules of use
• Usage
• User accounts
• Queue system
• Parallel jobs
• Available queues
• Available programs
• Accessing the nodes
• Latest news and changelog
• Comments

Very brief rules of use

• No interactive jobs allowed, use qsub.
• If possible, jobs should run on /wrk, not on /home.
• Only install programs in your home directory, even if you could install elsewhere. Do not modify anything else either. Otherwise cluster management will become impossible.

Usage

The front end will be busy enough as a file server and queue master, as well as test.q home.

See below for info on different queues and node access.

User accounts

Queue system

Job scripts

• To submit a job to the queue, you need to prepare a job script. The job script is like a normal script of commands with additional settings related to the queue, like which queue to use, where to report job status, etc.

• Just to get the idea, a very simple example job script could be the following:

      #!/bin/sh
      #$ -o outputfile.out 
      #$ -N NameOfJob 
      #$ -q test.q
  
      sleep 60
      echo "took a nap."
  

• This "job" of course does nothing useful. It does show the general idea of a job script, though: First comes a bunch of parameters (lines beginning with #$), then the commands that will be run once the queue system submits the job to some node for execution.

• The most important job parameters are:

  option	explanation
  -o	The file where the stdout of the job is directed.
  -j	stderr is combined with stdout.
  -N	The name of the job. It is a good idea to give your different jobs different names, it eases identification.
  -S	Which shell to use.
  -V	Copies the current environment to the job session.
  -cwd	Execute from current working directory.
  -m	Sends email at specific occasions. Common occasions are e (end of job) and b (beginning of job)
  -M	Your e-mail address.
  -q	Name of the queue. See below for a list of available queues.

• For a complete list of options/parameters, execute man qsub.

• A few example job scripts:
  • turbomole-job.cmd. Turbomole 5-7-1 example.
  • gaussian-job.cmd. Gaussian03 example.

• Both scripts do things a bit differently, but have one thing in common: The job files are first copied to the local /wrk disks of the nodes!
• This is very important! The /home directories are mounted over NFS and should not host calculations. In other words: It is forbidden to perform calculations in your /home directory! The exception to the rule is jobs in the test.q, which run on the front end.

• So copy your files somewhere under /wrk/users/$USER, change to that directory, perform your calculation, move the resulting files back to your /home directory (which is visible from all nodes). NOTE! You must delete the files from /wrk after your calculation!

• There is also no guarantee as of how long the files on /wrk will stay there. They will be deleted automatically after 90 days. If /wrk is getting full, files that don't belong to the running job on the node can be deleted without further warning.

Submitting jobs

• After preparing the job script, submit it to the queue with the qsub command, for example:
  • qsub turbomole-job.cmd

• To see your job statuses (queueing or running, etc.), use the qstat command. With qstat -f you get more info. With qstat -ext even more. And for the brave, there's qstat -f -ext -r, which already wants a console 184 columns wide... The qstat output is quite wide always, so a useful alias might be:
  • alias q='qstat | cut -c 1-102 | grep -E job-ID\|$USER\|---'

Parallel jobs

Magnesium supports parallel jobs via the MPICH message passing interface (version 1.2.6). LAM/MPI might be coming. A coexistence with MPICH is not without problems; SGE and LAM/MPI are not the best of friends.

SGE, the queueing system, is not optimal for managing parallel jobs, so there are a few things that you must use in your job script when running parallel jobs.

1. You need to specify that you want to run a parallel job with the -pe option. Below, <numprocs> defines how many CPU's you want to use. The maximum allowed is 6! Adhere. Also, use the -V switch.
   • #$ -V
   • #$ -pe mpich <numprocs>

2. You have to set the environment variable MPICH_PROCESS_GROUP to no:
   • export MPICH_PROCESS_GROUP=no

3. Optimise MPICH message transfer speed with:
   • export P4_SOCKBUFSIZE=128000
   
   The P4_SOCKBUFSIZE is not related to SGE, but will significantly improve performance if large packets are transferred. MPICH cannot be faster than raw TCP. The latency increases to 25 µs, and the maximum transfer speed drops to about 410 Mbps, see figure. Values larger than 128000 could perhaps be useful.

Many parallel program packages need a directory that is visible to all processes on the nodes used. If this is the case, the only alternative is to run at least partly on /home. But even if some files need to be accessible to all processes, others, like two electron integral files and such, might not be. Find out the situation for your program. In any case, it is forbidden to store 2e-integrals on /home! (The disk quota effectively takes care of this anyway, but it is still forbidden.) Minimize other disk access to /home as well! For example, don't dump MO files after every SCF iteration, etc.

Parallel Turbomole specific

• For Turbomole, there are three additional environment variables that have to be set in the job script:
  1. You need to tell Turbomole that you would like to run the parallel version with
     • export PARA_ARCH=MPI

  2. Turbomole wastes one CPU for keeping track of the parallel run. For this reason, you need to reserve one more CPU than what will be used for the actual calculation. In other words, the environment variable PARNODES needs to be set to an integer one less than the <numprocs> requested (see above):
     • export PARNODES=<numprocs-1>

  3. You have to specify the environment variable HOSTS_FILE, otherwise Turbomole will ignore what SGE tells it to use, and just start on whatever nodes it feels like:
     • export HOSTS_FILE=$TMPDIR/machines

• For more info on parallel control keywords for Turbomole, see the TM manual.

Example parallel job script:

• turbomole-parajob.cmd. Parallel Turbomole 5-7-1 example.

Available queues

• #$ -q chem-smallmem.q

The queues have different limits. Choose the one that suits your needs.

The test.q can be used for example to test if a job really would start with the job script you've meticulously prepared. It's also useful for diagnosis of programs that seem to crash right after start.

The jobs in test.q run on the front end, so you don't necessarily have to copy your files to a /wrk-dir. And if you do, you don't have to log onto a node to check the files produced.

You can choose to run in either chem-smallmem.q or chem-largemem.q by selecting any chem queue with a wildcard, like chem*, i.e.:

• #$ -q chem*

To see for example how many free slots the queues have available:

• qstat -g c

Available programs

• Turbomole 5-7-1 (serial and parallel)
• Gaussian03 (serial only)

Accessing the nodes

If you need to log on, use ssh mgXX where XX identifies the node and runs from 01–15. For example

• ssh mg07

NOTE! There is no need to log on to a node for following job output and such. For this, there exists a special command, dsh (for distributed shell, or dancer's shell). dsh runs the command you specify on a host you specify. You only need to tell dsh which node (machine) you want to run on with the -m parameter. An example:

• dsh -m mg01 "tail -f /wrk/users/borg/output.out"

You could of course do the same thing directly with ssh, if you prefer:

• ssh mg01 "tail -f /wrk/users/borg/output.out"

The advantage of dsh is that you can specify more than one machine to perform an action on. With the -c switch, your command is executed simultaneously on all specified machines. More info via man dsh.

There is a special parameter for executing a command on all nodes; -a. For example, if you would like to see the main processes on all nodes, you could execute one of these:

• dsh -a -M "export TERM=vt100; top -b -n 1 | grep -A 4 USER"
• dsh -a "export TERM=vt100; top -b -n 1 | grep $USER | head -4"

To quickly clean up (delete) all your files from all /wrk disks (note that the front end is not included with the -a switch):

• dsh -a -M -c "rm -rf /wrk/users/$USER/*"

For a list of all nodes as well as some info about them, you can use qhost. Withe the -j switch, it's an alternative to qstat:

• qhost -j

Latest news and changelog

• 02 Oct 2005: All 15 nodes up. Enabled printing from the front end.
• 30 Sep 2005: Updated "largemem" host info. Updated kernels to 2.6.12-1.
• 15 Jun 2005: MPICH parallel jobs enabled.
• 13 Jun 2005: Added chem-* queues. Updated kernels to 2.6.11-9 as 2.6.8-11 panics.
• 11 Jun 2005: All 11 nodes up.
• 10 Jun 2005: /home was converted from XFS to ext3 just in case.
• 09 Jun 2005: Winter hibernation ceases, Mg awakens with one node up.

Comments

Mg in the night