Oracle HPC (Legacy)

This Page is for the temporary / first iteration of the Oracle platform, which needed to be revised. It is for reference only as this system is deprecated.

System layout / architecture

The Biociphers Oracle HPC consists of one combined transfer / submit node (similar to both mercury and consign of PMACS HPC), which will be referred to here as the 'headnode'. This machine is the only one with inbound internet access, however, all storage is available from it, so you will never need to do any kind of ssh tunnel setup. You can check the status of the cluster and submit your jobs from this node. It is also acceptable to run your rsync or other data transfer tools from this node, but I may revise this if it turns into a performance bottleneck in the future.

Headnode performance: 8 CPUs, 64gb RAM

The rest of the system (the "computenodes") are split into three segments: "general purpose", "gpu", and "scratch".

-General purpose machines are 128 cores / 256GB ram each. There are currently 28 available.

-GPU machines have 24 cores / 64GB ram, 4 Tesla V100 GPUs with 16GB VRAM. There are currently 2 available.

-Scratch machines have 48 cores, 320GB RAM, 4x NVME storage chips in RAID0 to create a 24TB scratch space. There are currently 6 available.

Storage

Primary storage is provided by Oracle in increments of a maximum size of 32 TB. In order to accommodate prior LPC data, two of such volumes are available:

/data1

/data2

For those migrating from LPC, /data1 contains barash_hdb1,barash_hdb2, and barash_hdr3 , while /data2 contains barash_hdr1 and barash_hdr2.

Additional storage space can be added within a day or two on an ad-hoc basis.

Home directory storage under /home is currently of size 1TB shared. In general, you should only keep your own code and program installations in your home, and keep data under primary or scratch locations.

There is a platform third parts software location under /opt_hpc. This is a read only filesystem which will house non-packaged, stable versions of software such as STAR. You should contact the HPC administrator to add additional software to this location for usage.

Finally, scratch storage has its own usage. Each of the 'scratch' nodes, has one RAID0 volume mounted to itself under /scratch , which is also bound to a specific node names location such as /scratch001 or /scratch002 (on compnode-scratch001 and compnode-scratch002 respectively). These storage locations are mounted directly to each specific scratch node, and thus you will get the best speed by far only reading / writing to /scratch001 from the node 'compnode-scratch001', for example. However, for user convenience, these filesystems are also network mounted to the headnode as well as the general purpose compute nodes and GPU compute nodes. But be aware, the access using this method will be significantly slower than the direct usage on the node. For this reason, you should likely specify a specific scratch node to use when running a number of jobs in a row, so that you don't need to manually compile all of the results together in the end. Scratch nodes should be used for I/O heavy tasks such as MOCCASIN or majiq build using SJ files.

Access/Connection

Currently, the lab developer(s) (Paul Jewell, at time of writing), can grant you access to the system. This access will come in the form of a username, private key file, and public key file. You will use the private key file (which by default has NO file extension, and is the same name as your username) to log into the HPC.

The hostname of the node to log into is:

hpc.biociphers.org (132.145.35.68)

For example, using standard openssh:

$ ssh -i /path/to/keyfile username@hpc.biociphers.org

If you get a message along the lines of "unprotected key file.....", you will need to first run:

$ chmod 600 /path/to/keyfile

And try again.

Viewing and submitting jobs

The oracle HPC uses SGE "Sun Grid Engine" as it's grid management system. There are ample supplementary manuals available online for more details on the many options available for the grid viewing and submission commands. Similar to other batch managers, you can either create a bash script for your job, and submit that, or run interactively. Some simple use cases will be specified here:

Overview of available resources:

$ qstat -f

This will display all nodes and the available and used cpu slots on each. By default you will get detailed information about the jobs for your own user. If you would like to get details for the jobs ran by other users as well, you can run:

$ qstat -f -u '*'

Submit a batch job:

$ qsub -q general.q -pe general_comp 4 job.sh

$ qsub -q scratch.q -pe scratch_comp 4 job.sh

$ qsub -q gpu.q -pe gpu_comp 4 job.sh

Here, the switch "-q" specifies the queue to use, and "-pe" specifies the "parallel environment" to use, along with the number of parallel processes. In the current grid configuration (non MPI usage), all of the CPU cores requested will always be on the same machine, which means that the maximum realistic number of processes to use for a single job or interactive session is 128. (In this case, we request 4 cores for each batch job) For all job submission commands (batch or interactive) you must specify a queue with the -q switch, or your job will jump into a random queue.

Submitting an interactive session:

$ qrsh -q general.q

Uses the same switches for resources as the qsub command.

Request specific node:

qrsh -q scratch.q -pe scratch_comp 4 -l h=compnode-scratch001

This is useful for using the scratch node, so that you may consistently use the same storage space.

Please note: you may receive a message "biociphers groups: cannot find name for group ID 40000" or some similar, when using an interactive session. This is not an error! The gridengine uses dynamic groups for resource tracking, which are created on the fly. I've looked into this specific message, and there is no need to concern at all about it. Your user will still create and read files with the "biociphers" group by default.

Optionally, for any of these commands, you may specify the switch "-l h_vmem=2G" (for example) , to limit the job, per slot, to this amount of RAM. Currently, we are not mandating that this switch be used, but may in the future depending on how conflicting the usage will be.

Software installation

If a program is large, has many system dependencies, or would be useful equally to all users, please ask the system administrator for suggestions on it's installation, before trying to do the installation yourself.

In general, there is no issue with maintaining a virtualenv for majiq, moccasin, or other such frequently updated tools inside your home directory.