Skip to content

Gwave (PSU)

Key information
Home page
Account sign-up https://grouper.ligo.org/ldg/
Support iask@ics.psu.edu

Login hosts

Hostname Description Memory CPU model cores/CPU total threads
ligo01.gwave.ics.psu.edu Production submit node 192GB 2 x 2.6GHz Xeon Gold 6126 12 48
ligo02.gwave.ics.psu.edu Production submit node 192GB 2 x 2.6GHz Xeon Gold 6126 12 48

For details on how to connect to these machines, please see Access to the LIGO Data Grid.

Execute hosts

Hostname Memory CPU model cores/CPU total threads
comp-ex-0{001-184} 192GB 2 x 2.6GHz Xeon Gold 6126 12 48

Functional Architecture

GWave cluster functional architecture

Provided Services and Dashboards

Service URL
user webspace https://ldas-jobs.gwave.ics.psu.edu/~albert.einstein/
cluster utilization dashboards https://foreman01.ext.gwave.ics.psu.edu/grafana/dashboards
disk usage map https://foreman01.ext.gwave.ics.psu.edu/diskusage/
OSG jobs on GWave https://gracc.opensciencegrid.org/d/000000079/site-summary?orgId=1&var-site=PSU%20LIGO

User Information

Important Directories

  • /ligo/home/ligo.org/albert.einstein/: regular files, analysis output, etc.
  • /ligo/home/ligo.org/albert.einstein/public_html/: files viewable by LIGO-authtenticated users. See below for setting up file sharing
  • /ligo/software/ligo.org/albert.einstein/: for software builds
  • /cvmfs: CVMFS data files served from a local NFS Aliencache and remote Stashcaches

Data Access

Frame files are provided by CVMFS via repositories in the /cvmfs directory. To access proprietary data, you must first create an active proxy via ligo-proxy-init user.name. You must do this prior to sourcing environments or running analyses that require proprietary data.

Sharing LIGO Files

Files in your home directory may be shared by using the /ligo/home/ligo.org/albert.einstein/public_html/ folder. To do this, you must set up permissions for both public_html and your home folder:

chmod 775 /ligo/home/ligo.org/albert.einstein/public_html
chmod 711 /ligo/home/ligo.org/albert.einstein/

This allows your files to be viewable by LIGO-authenticated users at https://ldas-jobs.gwave.ics.psu.edu/~albert.einstein/ (replacing albert.einstein with your LIGO credentials).

Building Software

When compiling software, use the /ligo/software/ directory instead of the /ligo/home/ directory.

Conda Environments

By default, user conda environments are stored in /ligo/home/, so this must be changed via:

ENV_DIR='conda-envs'    # this is the folder name where you want to keep your environments
mkdir -p /ligo/software/ligo.org/${USER}/${ENV_DIR}
conda config --add envs_dirs /ligo/software/ligo.org/${USER}/${ENV_DIR}

Use conda activate to load environments as usual. Verify your setup with conda info --env; output should be similar to this:

$ conda info --env
# conda environments:
#
test-env             *  /ligo/software/ligo.org/albert.einstein/<conda-envs>/envs/test-env

To shorten the long path that prefixes the terminal prompt, enter:

conda config --set env_prompt '({name})'

After building software or moving environments to the /ligo/software directory, reference that path in your HTCondor submit and DAG files in place of your /ligo/home directory.

Other Tips

SSH keys

To SSH to GWave without entering your password each time, copy your public SSH key using:

ssh-copy-id -i ~/.ssh/<your_pubkey.pub> albert.einstein@ligo01.gwave.ics.psu.edu

You will be prompted for your password to copy the file. Afterwards, you will be able to SSH without re-entering your password.