Migrating a workflow off the shared home filesystem¶

This tutorial walks through converting a "legacy" HTCondor workflow — one that implicitly relies on the user's /home filesystem being mounted on the worker node — into a self-contained workflow that uses HTCondor file transfer and globally-accessible software. Such a workflow can run on the widest possible set of resources, including nodes that do not mount /home and distributed resources where a shared filesystem is infeasible.

For the background and policy driving this change, see Working without a shared filesystem and the roadmap item Disable mounting of home directories on LIGO Lab worker nodes.

Conventions in this tutorial

Examples assume submission from a LIGO Lab CIT access point, where the default token issuer is the access-point (AP) issuer; this is why the examples use use_oauth_services = scitokens and unprefixed osdf:// URLs. On an access point that uses the IGWN issuer, adjust the token syntax as described in SciTokens (use_oauth_services = igwn, igwn+osdf://, …). Replace the example username albert.einstein with your own throughout, and replace the placeholder accounting tag ligo.dev.o4.mygroup.mypipeline with the one for your own analysis (an accounting tag is mandatory for every job).

The starting point: a workflow that assumes `/home`¶

Consider a typical legacy submit file. Nothing in it tells HTCondor how to move data or software around, because everything lives under /home and the worker node used to mount it:

before.sub — implicitly depends on a mounted /home

# software: a bespoke Conda environment installed under /home
executable = /home/albert.einstein/.conda/envs/myenv/bin/python
arguments  = analyse.py --input /home/albert.einstein/data/input.h5 \
                        --output /home/albert.einstein/results/output.h5

log    = analyse.log
error  = analyse.err
output = analyse.out

request_cpus   = 1
request_disk   = 1GB
request_memory = 4GB

accounting_group = ligo.dev.o4.mygroup.mypipeline

queue

This job carries four hidden dependencies on the shared filesystem:

Dependency	Where it appears	Fails on a node without `/home` because…
Software	the `/home/.../myenv/bin/python` interpreter and its Conda environment	the interpreter and its libraries are not present on the execute node
The analysis script	`analyse.py` (implicitly read from the submit directory under `/home`)	the script is never transferred into the job
Input data	`--input /home/.../input.h5`	the input path does not exist on the execute node
Output data	`--output /home/.../results/output.h5`	the output directory does not exist, so the write fails

The remaining steps remove each dependency in turn.

Step 1 — Deliver the software globally¶

Replace the /home Conda environment with software that any execute node can reach. Choose the route that fits your dependencies (full detail in Accessing software):

It's all in the IGWN Conda DistributionYou need bespoke software (recommended for custom envs)

If every package you need is already in the IGWN Conda Distribution, you need not build or host anything: run your script through conda run straight from CVMFS.

executable = /cvmfs/software.igwn.org/conda/bin/conda
arguments  = run --name igwn --no-capture-output python analyse.py ...

For a custom environment — the usual case when you had packages installed under /home — build a SIF container image that contains your software and host it on OSDF:

apptainer build myenv-v1.sif myenv.def
pelican object put myenv-v1.sif \
    osdf:///igwn/cit/staging/albert.einstein/myenv-v1.sif

You then run it in HTCondor's container universe (used in the finished submit file below). See Hosting the image in OSDF for the full build-and-publish recipe, and the containerised software tutorial for more on writing definition files.

This tutorial follows the container route from here on, since it is the recommended replacement for a bespoke /home Conda environment.

Step 2 — Transfer the analysis script into the job¶

Because the job no longer runs from your /home directory, the analysis script must be transferred in explicitly with transfer_input_files. HTCondor copies it into the job's scratch directory, and the job refers to it by its bare local name:

transfer_input_files = analyse.py
arguments            = analyse.py --input input.h5 --output output.h5

Step 3 — Get the input data in¶

Stop reading the input from /home. For a modest file you can list it directly in transfer_input_files; for a large input (≳ 1 GB), first stage it into your /igwn/cit staging namespace and transfer it from OSDF:

# one-time: publish the input into your OSDF staging area
pelican object put input.h5 \
    osdf:///igwn/cit/staging/albert.einstein/input.h5

transfer_input_files = analyse.py, osdf:///igwn/cit/staging/albert.einstein/input.h5

Reading from the /igwn namespace requires a SciToken; on the AP issuer this is simply use_oauth_services = scitokens (see Using tokens with transfer_input_files).

Size request_disk for everything in the sandbox

Every transferred input — plus the container image and all output — lives in the job's scratch directory and counts towards request_disk. See request_disk.

Step 4 — Get the output data out¶

Stop writing results to /home. Have your job write output.h5 into its scratch directory (a bare relative path), then let HTCondor return it.

For small outputs, transfer_output_files copies the file back to your submit directory on the access point:

transfer_output_files = output.h5

For large outputs, or to send results back to CIT from a job running at a remote site, remap the file to your /igwn/cit staging area with transfer_output_remaps (this needs a token with the write:/staging scope):

transfer_output_files  = output.h5
transfer_output_remaps = "output.h5 = osdf:///igwn/cit/staging/albert.einstein/results/output-v1.h5"

Write to a new, versioned path each time

Files published into /igwn/cit/staging cannot be modified once cached by OSDF. Write output-v1.h5, output-v2.h5, … rather than overwriting.

Step 5 — Set the environment and requirements¶

An HTCondor job inherits no environment by default — HOME, PATH, and USER are unset. Running inside the container gives your software the environment baked into the image, so in the container route you rarely need to set anything extra. If your application does require specific variables, set them explicitly with environment; never reach for getenv = true, which is strongly discouraged.

Finally, declare the machine requirements needed to run a SIF image from OSDF:

requirements = HAS_SINGULARITY && SINGULARITY_CAN_USE_SIF

The finished workflow¶

Putting the steps together gives a submit file with no dependence on a shared filesystem:

after.sub — self-contained, file-transfer based

# software: a user-supplied container image hosted on OSDF
universe        = container
container_image = osdf:///igwn/cit/staging/albert.einstein/myenv-v1.sif

# the interpreter lives inside the image; the script is transferred in
executable        = /usr/bin/python3
transfer_executable = false
arguments         = analyse.py --input input.h5 --output output.h5

# data in and out, all via file transfer — nothing touches /home
should_transfer_files  = yes
transfer_input_files   = analyse.py, osdf:///igwn/cit/staging/albert.einstein/input.h5
transfer_output_files  = output.h5
transfer_output_remaps = "output.h5 = osdf:///igwn/cit/staging/albert.einstein/results/output-v1.h5"

# token for the restricted /igwn namespaces (AP issuer)
use_oauth_services = scitokens

# match a node that can run a SIF image
requirements = HAS_SINGULARITY && SINGULARITY_CAN_USE_SIF

log    = analyse.log
error  = analyse.err
output = analyse.out

request_cpus   = 1
request_disk   = 5GB
request_memory = 4GB

accounting_group = ligo.dev.o4.mygroup.mypipeline

queue

Compared with before.sub: the software now arrives as a container image from OSDF, the script and input are transferred into the sandbox, the output is transferred back out (here to OSDF staging), and every path the job uses at run time is a local name in the scratch directory rather than a /home path.

Set transfer_executable = false for an in-image executable

executable here names a program inside the container (/usr/bin/python3), not a file on the access point, so transfer_executable = false stops HTCondor trying to ship it from the submit host. See Running the image with HTCondor.

Test before scaling up¶

Submit a single job first and confirm it completes:

condor_submit after.sub

Watch its progress with the job monitoring tools; since you can no longer watch files update in /home, enable streaming of stdout/stderr if you want near-real-time output.
If the job goes on hold, read the hold reason:
```
condor_q -hold
```
A file-transfer error or a "no such file or directory" message that references a /home path points to a remaining dependency — revisit the step above that covers it (software, script, input, or output).
Once a single job succeeds on a node that does not mount /home, scale up with confidence.

Need a temporary bridge while you migrate?

If you must keep a not-yet-migrated workflow running in the meantime, you can restrict it to the shrinking set of nodes that still mount /home using the interim compatibility options (requirements = (TARGET.EPNFS =?= True), or the epnfs.ligo.caltech.edu access point). This is a bridge only — the aim of this tutorial is to remove the dependency altogether.