Accessing software with HTCondor¶
Given the IGWN grid jobs can run at any number of computing centres, including opporunistic resources not operated by IGWN members, job configurations cannot assume that the jobs will be matched with a resource that already has a local installation of any software or dependencies. Such software must be available from a globally-accessible repository or sent with the job using HTCondor's file transfer mechanism.
Summary of use cases¶
The following table lists some typical use-cases and suggested software deployment solutions:
Use-case | Recommendation |
---|---|
Production-level analysis code | IGWN conda environments (CVMFS) or CVMFs-deployed Singluarity container |
Developmental / pre-release code | User-supplied Singularity container (CVMFS) |
Statically compiled binary / shell script | Transfer executable with job |
Conda environments in CVMFS¶
An easy way to access software remotely is to add your package to the IGWN Conda Distribution, which is automatically published to the OASIS CVMFS repository.
For instructions on how to use executables from a Conda environment in an HTCondor workflow, please see Conda environments.
Singularity container images in CVMFS¶
A container is a unit of software that packages code and dependencies to provide a complete and extremely portable run-time environment. Singularity is one of several container technologies which is particularly well-suited to shared distributed computing environments.
A complete recipe for building and publishing containers to CVMFS can be found in Containerisation of software.
Once your Singularity container has been published to CVMFS:
-
configure your job submit file to use the path to the executable in the container,
executable = /usr/bin/mything
-
specify the path to the container image in CVMFS:
MY.SingularityImage = "/cvmfs/singularity.opensciencegrid.org/<org>/<image>:<tag>"
-
ensure that your job matches a host with the ability to run Singularity by specifying the
HAS_SINGULARITY
requirement:requirements = HAS_SINGULARITY=?=True
HTCondor job using a Singularity container in CVMFS
executable = /usr/bin/BayesWave
transfer_executable = False
MY.SingularityImage = "/cvmfs/singularity.opensciencegrid.org/lscsoft/bayeswave:v1.0.6"
requirements = (HAS_SINGULARITY=?=True)
log = example.log
error = example.err
output = example.out
request_disk = 1GB
request_memory = 16GB
accounting_group = ligo.dev.o5.cbc.pe.bayeswave
queue
Transferring Singularity images using HTCondor file transfer¶
Work in progress
This section is a work in progress, please consider contributing to complete this if you can.
Singularity images via Dockerhub¶
It is possible, but not generally recommended, to specify a docker URI for the singularity image. For example:
MY.SingularityImage = "docker://igwn/software"
In this case, HTCondor will invoke a singularity pull
command prior to starting your job, which will download and convert the Docker image to Singularity on the fly.
Limit use of Docker URIs for SingularityImage
Specifying MY.SingularityImage = "docker://some/image"
means every job in your workflow will attempt to download and convert that container image. This can quickly induce enormous loads on local worker nodes, as well as overwhelming local network bandwidth at the execute point. Please only use this option for very small scale development tests.
Standalone executables¶
If your software and all dependencies can be installed without administrator priveleges and is relatively platform-independent, an variation on this option is to use a script to install software on the fly at the start of job execution (but be aware that this will affect your jobs' run time). In all cases, be mindful of the bandwidth and disk space requirements for potentially large statically-compiled binaries and on-the-fly installations: software and data distributed through CVMFS uses smart caching mechanisms to make efficient use of bandwidth; installing packages as part of your job generally does not.
To use this method of software distribution, configure the job submit file with the path to your executable and tell HTCondor to transfer the executable with the job:
HTCondor submit file with executable transfer enabled
Specify the path to the executable
and enable file transfer:
executable = /home/albert.einstein/hello_world.sh
transfer_executable = true
log = example.log
error = example.err
output = example.out
request_disk = 1GB
request_memory = 1GB
accounting_group = ligo.dev.o5.compsoft.hello.world
queue