Skip to content

Working without a shared filesystem

Historically, jobs submitted to the LIGO Lab clusters (CIT, LHO, LLO) could rely on the user's /home filesystem being mounted, over NFS, on every worker (execute) node. This allowed jobs to read input data, write output data, and load software — for example a Conda environment or Python virtual environment installed under /home — directly from the shared filesystem, without telling HTCondor how to move any of it.

IGWN is moving away from this model. Requiring every worker node to mount every user's /home is unsustainable for the stable performance of those filesystems, and it is fundamentally incompatible with the wider pool of geographically-distributed resources — including opportunistic resources not operated by IGWN members — where a shared /home mount is infeasible. A job that does not depend on a shared filesystem can run on the widest possible set of resources.

This change is on the Computing Roadmap

The mounting of /home on LIGO Lab execute nodes is being disabled, starting at CIT and rolling out gradually to the other LIGO Lab clusters. Jobs that assume a mounted /home without declaring that requirement will fail on a growing fraction of worker nodes over time.

For the authoritative motivation, timeline, and affected sites, see the roadmap item Disable mounting of home directories on LIGO Lab worker nodes.

This affects execute nodes, not access points

The change applies to execute (worker) nodes only. The /home filesystem remains mounted on the access points (submit hosts), so you can still prepare, submit, and inspect your workflows from /home as before. What changes is that your running jobs can no longer assume /home is present on the machine they land on.

For a complete, worked conversion of a legacy workflow, see the tutorial Migrating a workflow off the shared home filesystem.

Migration checklist

Work through the following for each workflow. Each item links to the relevant detail, either below or on the dedicated feature pages.

  1. Stop loading software from /home. Do not point executable at, or activate, a Conda environment or virtual environment that lives under /home. Deliver software through a globally-accessible channel instead — see Software below and Accessing software.

  2. Transfer input data into the job. Do not read input files from a /home path. Declare every input with transfer_input_files, using OSDF for large or widely-shared data — see Input data below.

  3. Transfer output data out of the job. Do not write results to a /home path. Let HTCondor return outputs with transfer_output_files, remapping large outputs to OSDF user-curated staging with transfer_output_remaps.

  4. Set the job environment explicitly. The default job environment does not set HOME, PATH, or USER, and does not inherit your login profile. Set what your job needs with the environment submit command; do not use getenv = true — see Job environment below.

  5. Declare machine requirements. Make sure your requirements and request_disk/request_memory are set for the resources your job actually needs — including enough request_disk to hold every transferred input, the software image (if any), and all output.

  6. Test before scaling up. Submit a single job first and confirm it completes without touching /home. See Verifying below and Monitoring job status.

A short-term bridge exists, but is not the destination

While you migrate, you can keep running on the shrinking set of nodes that still mount /home using the interim compatibility options. Treat this only as a bridge: the pool of such nodes will decrease over time, and these jobs cannot reach distributed resources.

Delivering software without /home

Software installed in a Conda environment or Python virtual environment under /home will not be visible to a job on a worker node that does not mount /home. Replace it with one of the globally-accessible distribution methods documented in Accessing software:

Reusing an existing Conda environment

If you have an existing custom Conda environment you cannot easily rebuild as a container, you can relocate it with conda-pack: pack the environment into a tarball, transfer that tarball into the job with transfer_input_files, unpack it inside the job's scratch directory, and activate it from there. Building a container image is more robust and reproducible, but conda-pack can be a pragmatic first step off /home.

Getting input data into the job

Read every input through HTCondor file transfer rather than from a /home path. HTCondor copies the declared files into the job's scratch (sandbox) directory on the execute node, and your application reads them using their local path (the bare filename), not a /home path.

  • For small inputs local to the access point, list them directly in transfer_input_files.

  • For centrally-managed or shared IGWN data, transfer them from OSDF; discover the osdf:// URLs as described in Discovering OSDF URLs.

  • For your own large inputs (≳ 1 GB), stage them into the /igwn/cit user-curated namespace and transfer them from there, rather than repeatedly reading them out of the access point.

Access to the restricted /igwn namespaces requires a SciToken; see Using tokens with transfer_input_files.

Getting output data out of the job

Return results with HTCondor file transfer instead of writing them to /home:

  • Name the files your job produces in the job's scratch directory with transfer_output_files; HTCondor copies them back to your submit directory on the access point when the job completes.

  • For large outputs, or to send results back to CIT from a job running at a remote site, remap them to your /igwn/cit staging area with transfer_output_remaps, which requires a token with the write:/staging scope.

Files published into /igwn/cit/staging cannot be modified

OSDF serves files from a global network of caches that will not pick up an in-place change at the origin. Always write to a new, versioned path rather than overwriting. See Published files cannot be modified.

Setting the job environment

Without a shared /home, your job also loses the login-shell environment it may have implicitly depended on. By default an HTCondor job inherits no environment from the access point or your login profile — not even HOME, PATH, or USER.

  • Set the variables your job needs explicitly with the environment submit command.

  • If your software relies on variables normally set by conda activate, run it through conda run, which reproduces the activation, rather than depending on your interactive shell.

  • Do not use getenv = true to paper over a missing environment: it is strongly discouraged and may be disabled, and it couples your job to whatever happens to be set on the submit host.

Interim compatibility with /home-mounting nodes

Some LIGO Lab worker nodes still mount /home during the transition. If a workflow genuinely cannot yet be migrated, you can restrict it to those nodes in one of two ways. Both are temporary bridges: the set of /home-mounting nodes will shrink over time, and such jobs cannot run on the wider distributed pool.

Add the EPNFS machine attribute to your job requirements so it only matches nodes that still mount /home:

requirements = (TARGET.EPNFS =?= True)

Combine it with any other requirements your job needs using &&.

Submit through the dedicated CIT access point epnfs.ligo.caltech.edu, which automatically restricts jobs to nodes that mount /home, without your needing to edit requirements.

Plan to remove the dependency

Neither option is a long-term solution. As the roadmap rollout proceeds, jobs that require EPNFS will have access to steadily fewer resources. Use the bridge only to keep running while you work through the migration checklist.

Verifying a migrated workflow

Before scaling up, confirm your job no longer depends on /home:

  • Submit a single job and confirm it completes successfully — ideally on a node that does not mount /home (i.e. without the EPNFS bridge).

  • Check that no path in your submit file, executable, arguments, or environment points into /home on the execute side.

  • If a job goes on hold with a file-transfer or "no such file or directory" error referencing a /home path, that path is a remaining dependency to convert to file transfer.

  • Use the job monitoring tools to inspect progress and intermediate output, since you can no longer watch files update in /home.

See also