Accessing the IGWN Computing Grid¶
As described already, the IGWN Computing Grid is a distributed HTC platform that combines resources from many locations.
Resource pools¶
Collections of resources where you can run jobs in HTCondor are called "pools".
The IGWN Grid pool ('IGWN Pool') is the collection of resources that support centralised submission from an Access Point and distributed execution.
Additionally, many of the IGWN computing centres support a local resource pool for workflows that cannot be submitted to the IGWN Pool.
Try to use the IGWN Grid Pool by default
All IGWN workflows should attempt to use the IGWN Grid Pool by default, unless it has been demonstrated that their workflow cannot use the pool for technical reasons (including data access requirements).
If you need help in migrating an existing workflow to use the IGWN Grid Pool, or debugging why it doesn't work, please don't hesitate to Ask for help.
Access points (Submit hosts)¶
IGWN Grid Access Points¶
Workflows can be submitted to the IGWN Computing Grid (IGWN Pool) only from specific machines:
Workflows submitted from the following Access Points will run in the IGWN pool by default:
| Hostname | Location |
|---|---|
ldas-osg.ligo.caltech.edu | California Institute of Technology (LIGO Lab, USA) |
citlogin0.ligo.caltech.edu | California Institute of Technology (LIGO Lab, USA) |
citlogin1.ligo.caltech.edu | California Institute of Technology (LIGO Lab, USA) |
citlogin2.ligo.caltech.edu | California Institute of Technology (LIGO Lab, USA) |
citlogin3.ligo.caltech.edu | California Institute of Technology (LIGO Lab, USA) |
citlogin4.ligo.caltech.edu | California Institute of Technology (LIGO Lab, USA) |
citlogin5.ligo.caltech.edu | California Institute of Technology (LIGO Lab, USA) |
citlogin6.ligo.caltech.edu | California Institute of Technology (LIGO Lab, USA) |
ldas-osg.ligo-wa.caltech.edu | LIGO Hanford Observatory (USA) |
ldas-osg.ligo-la.caltech.edu | LIGO Livingston Observatory (USA) |
See here for documentation on how to access these machines.
Any computing task one wishes to run on the IGWN pool should be submitted from one of such submit hosts.
Local Access Points¶
In addition, each computing centre may host its own Access Point, with access only to the local resources pool. See the navigation links under Computing Centres for documentation specific to each computing centre.
Specifying which HTCondor pool to run in¶
Some IGWN Grid Access Points are configured such that jobs may opt-in to running in additional HTCondor pools. An example would be to run jobs in a local pool which is co-located with the Access Point and whose Execute Points can directly access the same storage as the Access Point.
When present, computing centers with IGWN Pool Access Points advertise their local pools with the following POOL_NAME attribute values:
| Location | Local POOL_NAME |
|---|---|
| California Institute of Technology (LIGO Lab, USA) | CIT |
| LIGO Hanford Observatory (USA) | LHO |
| LIGO Livingston Observatory (USA) | LLO |
To opt-in to running in a given pool in addition to the default IGWN pool, use the custom attribute MY.POOLS, e.g.:
MY.POOLS = "IGWN,CIT"
To restrict jobs to a specific pool, just specify that pool only, e.g.:
MY.POOLS = "CIT"
HTCondor submit file for jobs to run in the local Caltech pool only
Restrict jobs to the local Caltech pool with MY.POOLS = CIT:
To restrict jobs to running in the local Caltech pool:
executable = ./lalapps_somejob
MY.POOLS = "CIT"
log = example.log
error = example.err
output = example.out
request_disk = 10GB
request_memory = 6GB
accounting_tag = igwn.dev.o5.compsoft.lalapps.nobelwin
queue
Using the local pool can be a powerful option for workflows where most of the processing is done on the IGWN Grid Pool tool, but some jobs have special requirements, for example:
- pre-processing jobs - to extract data for further distribution using HTCondor file transfer and reduce the requirements for other jobs
- short-duration post-processing jobs such as output data aggregation or web page generation.
Meanwhile, opting-in to additional pools can help maximize access to resources in general and/or leverage features not present in the IGWN pool.
Which pools can I access from which Access Point?
At this time, IGWN Grid Access Points can only submit to the IGWN pool and to the co-located local pool. We plan to expand access in the future so that jobs can run anywhere we have access and users only need to specify actual job requirements.
| IGWN AP Location | Available pools |
|---|---|
| California Institute of Technology (LIGO Lab, USA) | IGWN,CIT |
| LIGO Hanford Observatory (USA) | IGWN,LHO |
| LIGO Livingston Observatory (USA) | IGWN,LLO |
MY.flock_local deprecated
Note that the MY.POOLS option supersedes the previous method for opting in to local pools with MY.flock_local and excluding the IGWN pool via MY.DESIRED_Sites="none". Those older options will remain supported for back-compatibility but users are advised to adopt MY.POOLS when possible.