Skip to content

Software deployment

You should assume IGWN Grid jobs will run at sites which do not have local installations of your, or anyone else's, analysis software or dependencies. The software must be available from a globally-accessible repository or sent with the job using HTCondor's file transfer mechanism.

The following table lists some typical use-cases and suggested software deployment solutions:

Use-case Recommendation
{: nowrap } Production-level analysis code IGWN conda environments (CVMFS)
{: nowrap } Developmental / pre-release code User-supplied Singularity container (CVMFS)
{: .nowrap } Statically compiled binary / shell script Transfer executable with job

Conda environments in CVMFS

An easy way to access software remotely is to add your package to the official IGWN conda distributions, which are automatically published to CVMFS.

CVMFS is the CernVM File System:

The CernVM File System provides a scalable, reliable and low-maintenance software distribution service. It was developed to assist High Energy Physics (HEP) collaborations to deploy software on the worldwide-distributed computing infrastructure used to run data processing applications. CernVM-FS is implemented as a POSIX read-only file system in user space (a FUSE module). Files and directories are hosted on standard web servers and mounted in the universal namespace /cvmfs.

To understand how to get your software into the IGWN conda environments, see: Software change requests with the SCCB

To use executables from the conda distributions:

  • configure the job submit file with the path to your executable
  • tell HTCondor not to transfer the executable, since it will already be available on the execution host:

HTCondor job for an application in the IGWN conda environment in CVMFS

Specify the path to the executable and disable file transfer:

universe = Vanilla
executable = /cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn-py37/bin/BayesWave
transfer_executable = False
log = example.log
error = example.err
output = example.out
queue 1

Notes:

  • IGWN grid jobs should always run in the Vanilla Universe (the HTCondor "Grid" Universe is for something else).
  • Specify the absolute path to the executable in CVMFS.
  • No special requirements! All jobs on the IGWN Grid have access to this CVMFS repository.

Singularity containers in CVMFS

In cases where an application is at a developmental pre-release stage (e.g. a personal fork of LALSuite), or where the dependencies cannot for some reason be satisfied by conda you should use Singularity containers.

A container is a unit of software that packages code and dependencies to provide a complete and extremely portable run-time environment. Singularity is one of several container technologies which is particularly well-suited to shared distributed computing environments.

A complete recipe for building and publishing containers to CVMFS can be found in Containerisation of software.

Once your Singularity container has been published to CVMFS, configure your job submit file to use the path to the executable in the container, specify the path to the container image in CVMFS, and ensure that your job matches a host with the ability to run Singularity:

HTCondor job using a Singularity container in CVMFS

Specify the path to the executable, Singularity container, disable executable file transfer and require Singularity on the worker node:

universe = Vanilla
executable = /usr/bin/BayesWave
transfer_executable = False
+SingularityImage = "/cvmfs/singularity.opensciencegrid.org/lscsoft/bayeswave:v1.0.6"
requirements = (HAS_SINGULARITY=?=True)
log = example.log
error = example.err
output = example.out
queue 1

Notes:

  • It is highly recommended that you allow the workflow management system to handle execution of your application in Singularity: do not attempt to execute Singularity yourself.
  • Path to the executable must be the path inside the container.
  • Disable HTCondor default behavior to transfer the executable with the job since the executable now lives inside the container image.
  • Use double-quotes to specify the absolute path to the Singularity Image in CVMFS.
  • Demand the job lands on an execute host with the ability to run Singularity. Additional requirements expressions can be added by enclosing each expression in parentheses and joining with logical operators, E.g., (HAS_SINGULARITY=?=True) && (HAS_LIGO_FRAMES=?=True)

Standalone executables

A more historical way to run jobs on grid resources is via statically-compiled binaries or simple scripts which can be sent to the execution host via HTCondor file transfer.

If your software and all dependencies can be installed without administrator priveleges and is relatively platform-independent, an variation on this option is to use a script to install software on the fly at the start of job execution (but be aware that this will affect your jobs' run time). In all cases, be mindful of the bandwidth and disk space requirements for potentially large statically-compiled binaries and on-the-fly installations: software and data distributed through CVMFS uses smart caching mechanisms to make efficient use of bandwidth; installing packages as part of your job generally does not.

To use this method of software distribution, configure the job submit file with the path to your executable and tell HTCondor to transfer the executable with the job:

HTCondor submit file with executable transfer enabled

Specify the path to the executable and enable file transfer:

universe = Vanilla
executable = /home/albert.einstein/hello_world.sh
transfer_executable = True
log = example.log
error = example.err
output = example.out
queue 1

Notes:

  • The default behavior for IGWN grid jobs is in fact to transfer the executable anyway but this can useful to add for transparency.
  • HTCondor will copy hello_world.sh to the job sandbox on the worker node where it will be renamed condor_exe.