Gwave (PSU)¶
| Key information | |
|---|---|
| Home page | |
| Account sign-up | https://grouper.ligo.org/ldg/ |
| Support | iask@ics.psu.edu |
Servers¶
| Hostname | Description | Memory | CPU model | cores/CPU | total threads |
|---|---|---|---|---|---|
{: .nowrap } ligo0{1,2}.gwave.ics.psu.edu | Production submit nodes | 192GB | 2 x 2.6GHz Xeon Gold 6126 | 12 | 48 |
{: .nowrap } comp-ex-0{001-184} | Production compute nodes | 192GB | 2 x 2.6GHz Xeon Gold 6126 | 12 | 48 |
Functional Architecture¶
Provided Services and Dashboards¶
| Service | URL |
|---|---|
| user webspace | https://ldas-jobs.gwave.ics.psu.edu/~albert.einstein/ |
| cluster utilization dashboards | https://foreman01.ext.gwave.ics.psu.edu/grafana/dashboards |
| disk usage map | https://foreman01.ext.gwave.ics.psu.edu/diskusage/ |
| OSG jobs on GWave | https://gracc.opensciencegrid.org/d/000000079/site-summary?orgId=1&var-site=PSU%20LIGO |
User Information¶
Important Directories¶
/ligo/home/ligo.org/albert.einstein/: regular files, analysis output, etc./ligo/home/ligo.org/albert.einstein/public_html/: files viewable by LIGO-authtenticated users. See below for setting up file sharing/ligo/software/ligo.org/albert.einstein/: for software builds/cvmfs: CVMFS data files served from a local NFS Aliencache and remote Stashcaches
Data Access¶
Frame files are provided by CVMFS via repositories in the /cvmfs directory. To access proprietary data, you must first create an active proxy via ligo-proxy-init user.name. You must do this prior to sourcing environments or running analyses that require proprietary data.
Sharing LIGO Files¶
Files in your home directory may be shared by using the /ligo/home/ligo.org/albert.einstein/public_html/ folder. To do this, you must set up permissions for both public_html and your home folder:
chmod 775 /ligo/home/ligo.org/albert.einstein/public_html
chmod 711 /ligo/home/ligo.org/albert.einstein/
This allows your files to be viewable by LIGO-authenticated users at https://ldas-jobs.gwave.ics.psu.edu/~albert.einstein/ (replacing albert.einstein with your LIGO credentials).
Building Software¶
When compiling software, use the /ligo/software/ directory instead of the /ligo/home/ directory.
Conda Environments
By default, user conda environments are stored in /ligo/home/, so this must be changed via:
ENV_DIR='conda-envs' # this is the folder name where you want to keep your environments
mkdir -p /ligo/software/ligo.org/${USER}/${ENV_DIR}
conda config --add envs_dirs /ligo/software/ligo.org/${USER}/${ENV_DIR}
Use conda activate to load environments as usual. Verify your setup with conda info --env; output should be similar to this:
$ conda info --env
# conda environments:
#
test-env * /ligo/software/ligo.org/albert.einstein/<conda-envs>/envs/test-env
To shorten the long path that prefixes the terminal prompt, enter:
conda config --set env_prompt '({name})'
After building software or moving environments to the /ligo/software directory, reference that path in your HTCondor submit and DAG files in place of your /ligo/home directory.
Other Tips¶
SSH keys¶
To SSH to GWave without entering your password each time, copy your public SSH key using:
ssh-copy-id -i ~/.ssh/<your_pubkey.pub> albert.einstein@ligo01.gwave.ics.psu.edu
You will be prompted for your password to copy the file. Afterwards, you will be able to SSH without re-entering your password.