High Performance Computing Service

Filesystems

Summary of available filesystems

DirectoryExplanationBacked up?Disk quotas? Files removed?
/home/<username>/  User home directory*Yes (daily)Yes (40GB)No
/scratch/<username>/  Scratch user directories*NoNoNo automatic deletion currently
/local/  Per-node local disk**NoLimited by partition size (33GB or 108GB)Deleted after job completion
/ramdisks/  Per-node tmpfs (in-memory)**NoLimited by physical RAM (3.9GB or 18GB)Deleted after job completion
/usr/local/ Cluster-wide software YesN/A (not for user storage)N/A (not for user storage)

*These are shared Lustre filesystems - files created on these are immediately visible to all nodes.

**These are private for each compute node - files created on these are destroyed after completion of the current job.

Quotas

Each user has a limit on the amount of file storage he/she can use in the home directory, via quotas. Quotas on /home are currently set at 40 GB per user, although in practice one can exceed this by a few GB for a short amount of time. Use the command quota to list your current usage - this works for both /home and /scratch although the latter currently has no quotas enforced. Current usages appear under the kbytes heading (in kilobytes, 1GB=1024*1024KB). A * symbol next to this value indicates that you have exceeded your quota (listed in kilobytes under quota), and will need to reduce usage as soon as possible to below the quota value. Note that if the over-quota condition is not corrected after 7 days, creation of new files and directories under /home will become impossible, and if at any time usage reaches the hard limit given under limit the same will happen.

At the time of writing there are no quotas in force on /scratch, however this is likely to change following increased pressure on storage and the behaviour of some users.

Backups

At the time of writing, tape backups of data on /home are being made daily. Note that the number of backups retained which are applicable to a given period falls off rapidly with time and thus we cannot always retrieve deleted files (particularly short-lived files). Hence please be careful when deleting! If you need to request retrieval of a deleted file from backup, please email support'at'hpc.cam.ac.uk, stating the full pathname, and as closely as possible, when the file was deleted, when the file was last modified and when the file was created.

No backups are made of data in the scratch directories.

The /usr/local directory

A number of system-wide applications and libraries are installed under /usr/local/. There are several important subdirectories:

/usr/local/Cluster-Apps

This directory contains the cluster applications and libraries. Each application is installed under its own directory & version, and usually has an 'Environment Module' file. Modules are used for setting up compile-time and run-time environments conveniently, and for switching between different versions of the same application/library in a flexible way.

/usr/local/Cluster-Config

This directory contains module files, licenses for cluster software (e.g. Intel compilers), and configuration files for specific cluster software packages (e.g. scheduler software).

/usr/local/Cluster-Docs/

This contains documentation for certain cluster software packages.

/usr/local/Cluster-Users

Use of this directory, which is inherited from the ClusterVision integration, is deprecated. If you wish some software to be made available centrally, please email support'at'hpc.cam.ac.uk. Please see the current list of centrally installed software.