Skip to content

LDAS @ Caltech

Key information
Home page
Account sign-up See Requesting an account on the LDG page
Support ldas_admin_cit@ligo.caltech.edu
SSH CA Cert @cert-authority * ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHa03AZF3CvJ1C4Po15swSaMYI4kPszyBH/uOKHQYvu+EpehSfMZMaX5D7pUpc5cAXvMEEFzlZJQH4pOioIlqyE= IGWN_CIT_SSH_CERT_AUTHORITY

This service is available for all @LIGO.ORG registered collaboration members with accounts and access coordinated as part of the LIGO Data Grid.

Login hosts

Hostname Description Mem CPU model Core GPU
ldas-grid.ligo.caltech.edu Production submit 64G AMD EPYC 7402 VM 16
ldas-osg.ligo.caltech.edu IGWN OSG submit machine 196G Intel Gold 6136 24
ldas-pcdev1.ligo.caltech.edu Large mem/post-processing 1T AMD EPYC 7H12 128
ldas-pcdev2.ligo.caltech.edu GPU mixed 256G AMD EPYC 7502P 32 4
ldas-pcdev3.ligo.caltech.edu GPU mixed 128G Intel E5-2670 16 3
ldas-pcdev4.ligo.caltech.edu LSCSOFT update testing 128G Intel E5-2640 v3 16
ldas-pcdev5.ligo.caltech.edu Post-processing 512G Intel E5-2698 v4 40
ldas-pcdev6.ligo.caltech.edu Large mem/Post-processing 1.5T Intel Gold 6154 72
ldas-pcdev8.ligo.caltech.edu GPU TITAN V/optimized LSCSOFT RPM 96G Intel Gold 5115 10 1
ldas-pcdev10.ligo.caltech.edu Skylake benchmarking 64G Intel E3-1240 v5 4
ldas-pcdev11.ligo.caltech.edu GPU mixed 128G Intel E5-2630 v4 20 4
ldas-pcdev12.ligo.caltech.edu GPU 4 x A100-SXM4-80GB 512G AMD EPYC 7763 128 4
ldas-pcdev13.ligo.caltech.edu GPU mixed 128G Intel E5-2650 v4 24 4
ldas-pcdev14.ligo.caltech.edu Large mem benchmarking 256G AMD Opteron 6376 16
ldas-pcdev15.ligo.caltech.edu Rocky Linux 8.4 4G AMD EPYC 7402 2
cbc.ligo.caltech.edu Dedicated CBC 64G Intel E5-2630 v3 16
cwb.ligo.caltech.edu Dedicated cWB 256G Intel E5-2650 v4 24
detchar.ligo.caltech.edu Dedicated DetChar 128G Intel E5-2630 v3 16
emfollow.ligo.caltech.edu Dedicated EM Follow-up 64G AMD EPYC 7402 VM 16
gstlal.ligo.caltech.edu Dedicated GstLAL 32G AMD EPYC 7402 VM 8
lowlatency.ligo.caltech.edu Dedicated low-latency 64G Intel E5-2630 v3 16
spiir.ligo.caltech.edu Dedicated SPIIR 32G AMD EPYC 7402 VM 8

For details on how to connect to these machines, please see Access to the LIGO Data Grid.

Additional services

Service URL
JupyterLab https://jupyter.ligo.caltech.edu
User webspace https://ldas-jobs.ligo.caltech.edu/~USER/

Configuring your user environment on LDAS

This page describes the default user environments on LDAS, and how to customise availability and versions of the following software distributions:

Intel oneAPI

The Intel oneAPI Base Toolkit is available by default on LDAS, with the exception of the intelpython and mpi modules.

Disabling all Intel modules

To disable loading of all Intel oneAPI modules, create an empty file in your home directory called ~/.nointel:

touch ~/.nointel

Customising the included oneAPI modules

To take full control over which modules to include/exclude (including pinning specific versions) please create ~/.oneapi_config.txt that takes precedence over the default /opt/intel/oneapi/oneapi_config.txt

MATLAB

Enabling MATLAB

MATLAB is available on the command path by default, and can be discovered using which:

$ which matlab
/ldcg/matlab_r2015a/bin/matlab

Note

The default matlab version will be updated from time-to-time according to approval from the Software Change Control Board.

Enabling a specific version of MATLAB

To select a specific version of MATLAB, create a file in your ${HOME} directory named .usematlab_{release}, where {release} is the release number of MATLAB that you want, e.g:

touch ~/.usematlab_r2019a

Listing available MATLAB releases

To list the available MATLAB releases, just run this:

ls /ldcg/ | grep matlab

Disabling MATLAB

Disabling MATLAB

To opt out of all MATLAB releases, create a file in your ${HOME} directory named .nomatlab:

touch ~/.nomatlab

Warning

~/.nomatlab takes precedence over any ~/.usematlab_* files, so if you want to opt in after previously opting out, make sure and remove the old ~/.nomatlab file.

Restoring accidentally deleted/modified files at CIT

Home directories on the CIT cluster use the ZFS filesystem, which allows for periodic snapshots. This allows you to recover accidentally deleted/modified files as long as the file you want to recover existed when the snapshot was taken.

Here's an example. Let's assume you're a user working in your home directory on the CIT cluster:

$ pwd
/home/albert.einstein/temp
$ ls -l test.file
-rw-------   1 albert.einstein albert.einstein  8416 Feb 18  2020 test.file
$ rm test.file

Oops--you didn't mean to delete test.file!

To find out if you can recover this file, first you need to see what snapshots are available. You can find them by looking at the files in the .zfs/snapshot directory inside your home directory:

$ ls /home/albert.einstein/.zfs/snapshot
autosnap_2021-10-01_00:00:01_monthly/  autosnap_2021-12-07_00:42:36_weekly/  autosnap_2021-12-07_15:25:30_hourly/  autosnap_2021-12-08_05:56:38_hourly/
autosnap_2021-11-01_19:27:51_monthly/  autosnap_2021-12-07_02:43:53_hourly/  autosnap_2021-12-07_16:16:33_hourly/  autosnap_2021-12-08_07:03:20_hourly/
autosnap_2021-11-15_23:30:55_weekly/   autosnap_2021-12-07_03:32:00_hourly/  autosnap_2021-12-07_17:35:32_hourly/  autosnap_2021-12-08_08:34:30_hourly/
autosnap_2021-11-16_16:17:41_monthly/  autosnap_2021-12-07_04:18:34_hourly/  autosnap_2021-12-07_18:42:37_hourly/  autosnap_2021-12-08_09:22:19_hourly/
autosnap_2021-11-16_16:17:41_weekly/   autosnap_2021-12-07_05:33:12_hourly/  autosnap_2021-12-07_19:06:09_hourly/  autosnap_2021-12-08_10:32:40_hourly/
autosnap_2021-11-22_23:48:55_weekly/   autosnap_2021-12-07_06:06:00_hourly/  autosnap_2021-12-07_20:38:54_hourly/  autosnap_2021-12-08_11:52:12_hourly/
autosnap_2021-11-29_23:43:30_weekly/   autosnap_2021-12-07_07:42:33_hourly/  autosnap_2021-12-07_21:05:52_hourly/  autosnap_2021-12-08_13:02:57_hourly/
autosnap_2021-12-01_00:24:05_monthly/  autosnap_2021-12-07_08:07:01_hourly/  autosnap_2021-12-07_22:31:05_hourly/  autosnap_2021-12-08_14:48:35_hourly/
autosnap_2021-12-02_00:17:00_daily/    autosnap_2021-12-07_09:45:35_hourly/  autosnap_2021-12-07_23:47:03_hourly/  autosnap_2021-12-08_16:08:23_hourly/
autosnap_2021-12-03_00:11:47_daily/    autosnap_2021-12-07_10:15:41_hourly/  autosnap_2021-12-08_00:21:21_daily/   autosnap_2021-12-08_17:28:41_hourly/
autosnap_2021-12-04_00:04:44_daily/    autosnap_2021-12-07_11:47:58_hourly/  autosnap_2021-12-08_00:21:21_hourly/  autosnap_2021-12-08_18:28:45_hourly/
autosnap_2021-12-05_00:06:23_daily/    autosnap_2021-12-07_12:25:30_hourly/  autosnap_2021-12-08_02:14:08_hourly/  autosnap_2021-12-08_19:39:26_hourly/
autosnap_2021-12-06_00:12:35_daily/    autosnap_2021-12-07_13:11:19_hourly/  autosnap_2021-12-08_03:54:22_hourly/
autosnap_2021-12-07_00:42:36_daily/    autosnap_2021-12-07_14:44:16_hourly/  autosnap_2021-12-08_04:39:22_hourly/

As you can see, there are snapshots labeled with their time day and time (denoted in yyyy-mm-dd_hh:dd:ss format). To see if a particular file exists in a snapshot, you can just ls the snapshot for it. However, you should note that the <homedir>/.zfs/snapshot/<snapshot> will be the root of that snapshot, i.e. a picture of what <homedir> looked like at the time the snapshot was taken. Therefore, you'll need to look down the path where the file of interest lived (in the example, /temp):

$ ls -l /home/albert.einstein/.zfs/snapshot/autosnap_2021-12-08_11:52:12_hourly/temp/test.file
-rw------- 1 albert.einstein albert.einstein 8416 Feb 18  2020 /home/albert.einstein/.zfs/snapshot/autosnap_2021-12-08_11:52:12_hourly/temp/test.file

You can view/open any of the files in a snapshot just as you would with the original file, so you can check that the file is the version you want. However, please note that the snapshots are read only, so you cannot modify the file inside the snapshot.

Once you've found a version of the file to restore, you simply need to copy it back to your home directory so you can work with it. Simply use cp:

$ cp -ip /home/albert.einstein/.zfs/snapshot/autosnap_2021-12-08_11:52:12_hourly/temp/test.file /home/albert.einstein/temp
$ ls -l test.file
-rw------- 1 albert.einstein albert.einstein 8416 Feb 18  2020 test.file

The -p option will preserve the ownership and timestamps of the file (if that's what you want).

If you want to restore an entire directory tree, this is also possible, just use something like

$ cp -ipr /home/albert.einstein/.zfs/snapshot/autosnap_2021-12-08_11:52:12_hourly/temp /home/albert.einstein

to restore the entire "temp" directory, where the '-r' option is for a recursive copy of the entire tree.