Skip to content

Accessing software with HTCondor

Given the IGWN grid jobs can run at any number of computing centres, including opporunistic resources not operated by IGWN members, job configurations cannot assume that the jobs will be matched with a resource that already has a local installation of any software or dependencies. Such software must be available from a globally-accessible repository or sent with the job using HTCondor's file transfer mechanism.

Summary of use cases

The following table lists some typical use-cases and suggested software deployment solutions:

Use-case Recommendation
Production-level analysis code IGWN conda environments (CVMFS) or CVMFs-deployed Singluarity container
Developmental / pre-release code User-supplied Singularity container (CVMFS)
Statically compiled binary / shell script Transfer executable with job

Conda environments in CVMFS

An easy way to access software remotely is to add your package to the IGWN Conda Distribution, which are automatically published to the OASIS CVMFS repository.

To use executables from the conda distributions:

  • configure the job submit file with the full path to your executable:

    executable = /cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn/bin/python
    
  • tell HTCondor not to transfer the executable, since it will already be available on the execution host (and relocating it may break library links):

    transfer_executable = false
    

HTCondor job for an application in the IGWN conda environment in CVMFS

Specify the path to the executable and disable file transfer:

executable = /cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn/bin/BayesWave
transfer_executable = False

log = example.log
error = example.err
output = example.out

request_disk = 1GB
request_memory = 16GB

accounting_group = igwn.dev.o5.cbc.pe.bayeswave

queue

Singularity containers in CVMFS

A container is a unit of software that packages code and dependencies to provide a complete and extremely portable run-time environment. Singularity is one of several container technologies which is particularly well-suited to shared distributed computing environments.

A complete recipe for building and publishing containers to CVMFS can be found in Containerisation of software.

Once your Singularity container has been published to CVMFS:

  • configure your job submit file to use the path to the executable in the container,

    executable = /usr/bin/mything
    
  • specify the path to the container image in CVMFS:

    +SingularityImage = "/cvmfs/singularity.opensciencegrid.org/<org>/<image>:<tag>"
    
  • ensure that your job matches a host with the ability to run Singularity by specifying the HAS_SINGULARITY requirement:

    requirements = HAS_SINGULARITY=?=True
    

HTCondor job using a Singularity container in CVMFS

executable = /usr/bin/BayesWave
transfer_executable = False

+SingularityImage = "/cvmfs/singularity.opensciencegrid.org/lscsoft/bayeswave:v1.0.6"
requirements = (HAS_SINGULARITY=?=True)

log = example.log
error = example.err
output = example.out

request_disk = 1GB
request_memory = 16GB

accounting_group = ligo.dev.o5.cbc.pe.bayeswave

queue

Transferring Singularity images using HTCondor file transfer

Work in progress

This section is a work in progress, please consider contributing to complete this if you can.

Standalone executables

If your software and all dependencies can be installed without administrator priveleges and is relatively platform-independent, an variation on this option is to use a script to install software on the fly at the start of job execution (but be aware that this will affect your jobs' run time). In all cases, be mindful of the bandwidth and disk space requirements for potentially large statically-compiled binaries and on-the-fly installations: software and data distributed through CVMFS uses smart caching mechanisms to make efficient use of bandwidth; installing packages as part of your job generally does not.

To use this method of software distribution, configure the job submit file with the path to your executable and tell HTCondor to transfer the executable with the job:

HTCondor submit file with executable transfer enabled

Specify the path to the executable and enable file transfer:

executable = /home/albert.einstein/hello_world.sh
transfer_executable = true

log = example.log
error = example.err
output = example.out

request_disk = 1GB
request_memory = 1GB

accounting_group = ligo.dev.o5.compsoft.hello.world

queue