You should assume IGWN Grid jobs will run at sites which do not have local installations of your, or anyone else's, analysis software or dependencies. The software must be available from a globally-accessible repository or sent with the job using HTCondor's file transfer mechanism.
The following table lists some typical use-cases and suggested software deployment solutions:
|Production-level analysis code||IGWN conda environments (CVMFS)|
|Developmental / pre-release code||User-supplied Singularity container (CVMFS)|
|Statically compiled binary / shell script||Transfer executable with job|
Conda environments in CVMFS¶
To understand how to get your software into the IGWN Conda Distribution, see: Software change requests with the SCCB.
To use executables from the conda distributions:
- configure the job submit file with the full path to your executable
- tell HTCondor not to transfer the executable, since it will already be available on the execution host:
HTCondor job for an application in the IGWN conda environment in CVMFS
Specify the path to the
executable and disable file transfer:
universe = Vanilla executable = /cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn-py37/bin/BayesWave transfer_executable = False log = example.log error = example.err output = example.out queue 1
- IGWN grid jobs should always run in the Vanilla Universe (the HTCondor "Grid" Universe is for something else).
- Specify the absolute path to the executable in CVMFS.
- No special requirements! All jobs on the IGWN Grid have access to this CVMFS repository.
Singularity containers in CVMFS¶
In cases where an application is at a developmental pre-release stage (e.g. a personal fork of LALSuite), or where the dependencies cannot for some reason be satisfied by conda you should use Singularity containers.
A container is a unit of software that packages code and dependencies to provide a complete and extremely portable run-time environment. Singularity is one of several container technologies which is particularly well-suited to shared distributed computing environments.
A complete recipe for building and publishing containers to CVMFS can be found in Containerisation of software.
Once your Singularity container has been published to CVMFS, configure your job submit file to use the path to the executable in the container, specify the path to the container image in CVMFS, and ensure that your job matches a host with the ability to run Singularity:
HTCondor job using a Singularity container in CVMFS
Specify the path to the
executable, Singularity container, disable executable file transfer and require Singularity on the worker node:
universe = Vanilla executable = /usr/bin/BayesWave transfer_executable = False +SingularityImage = "/cvmfs/singularity.opensciencegrid.org/lscsoft/bayeswave:v1.0.6" requirements = (HAS_SINGULARITY=?=True) log = example.log error = example.err output = example.out queue 1
- It is highly recommended that you allow the workflow management system to handle execution of your application in Singularity: do not attempt to execute Singularity yourself.
- Path to the
executablemust be the path inside the container.
- Disable HTCondor default behavior to transfer the executable with the job since the executable now lives inside the container image.
- Use double-quotes to specify the absolute path to the Singularity Image in CVMFS.
- Demand the job lands on an execute host with the ability to run Singularity. Additional requirements expressions can be added by enclosing each expression in parentheses and joining with logical operators, E.g.,
(HAS_SINGULARITY=?=True) && (HAS_LIGO_FRAMES=?=True)
A more historical way to run jobs on grid resources is via statically-compiled binaries or simple scripts which can be sent to the execution host via HTCondor file transfer.
If your software and all dependencies can be installed without administrator priveleges and is relatively platform-independent, an variation on this option is to use a script to install software on the fly at the start of job execution (but be aware that this will affect your jobs' run time). In all cases, be mindful of the bandwidth and disk space requirements for potentially large statically-compiled binaries and on-the-fly installations: software and data distributed through CVMFS uses smart caching mechanisms to make efficient use of bandwidth; installing packages as part of your job generally does not.
To use this method of software distribution, configure the job submit file with the path to your executable and tell HTCondor to transfer the executable with the job:
HTCondor submit file with executable transfer enabled
Specify the path to the
executable and enable file transfer:
universe = Vanilla executable = /home/albert.einstein/hello_world.sh transfer_executable = True log = example.log error = example.err output = example.out queue 1
- The default behavior for IGWN grid jobs is in fact to transfer the executable anyway but this can useful to add for transparency.
- HTCondor will copy
hello_world.shto the job sandbox on the worker node where it will be renamed