Packaging software with conda¶

This document contains a brief primer on packaging software for distribution with conda.

What is a recipe?¶

A recipe is a collection of files that define how to build a Conda package. Each recipe minimally contains a meta.yaml file that describes the package name and version, its dependencies, and how to build it.

`meta.yaml`¶

For a fairly exhaustive description of what you can put in the meta.yaml file, see Defining metadata (meta.yaml) on the conda-build documentation.

`build.sh`¶

You can either define the build script inline in the meta.yaml file (example), otherwise conda-build will look for a build.sh script in the same directory (bld.bat for Windows). The build.sh script is a shell script that can do whatever you need it to do (example).

Packaging something new for conda-forge¶

All new recipes should be submitted as a Pull Request to the conda-forge/staged-recipes repository on GitHub. The basic instructions are (from here):

create a new recipe in a sub-directory of the /recipes directory in staged-recipes,
commit this new recipe to a branch of your fork, and propose a Pull Request
fix the inevitable linting and CI issues

Recipes can look quite different between languages, here are some good examples from IGWN Conda (feel free to suggest your own):

Python (using Pip as the build tool): gwpy,
C/C++ (Cmake): nds2-client,
Multiple outputs (Autotools): lal.

Maintaining a conda-forge recipe on GitHub¶

Once a recipe has been reviewed and merged, a new feedstock repository will be created to house the recipe, and all build and continuous integration (CI) utilities required to automatically build and upload packages. All changes to the recipe from that point should be made via Pull Requests against the feedstock repository.

There are automatic bots that will post Pull Requests whenever they think that the recipe is out of date. This is mainly done by searching for tarballs in the same location as the current release tarball that suggest a newer version has been uploaded.

Testing package builds¶

Basics¶

In principle, testing a package build should be as simple as installing conda-build and then executing conda build:

cd /path/to/recipe/parent/dir
conda build recipe/

e.g:

git clone https://github.com/conda-forge/lal-feedstock.git
cd lal-feedstock
conda build recipe

Building a feedstock recipe¶

If the recipe is already part of a feedstock, you should augment the above conda build command my passing the relevant configuration file for the target platform. These files live in the .ci_support/ directory of the feedstock, and on Linux for most projects that configuration file is called linux_64_.yaml, but for Python-based projects and other platforms the names are subtly different. The conda build command then becomes:

conda build recipe -m .ci_support/linux_64_.yaml

This configuration file pins a number of widely-utilised dependencies and better reproduces the official CI builds that are uploaded to anaconda.org.

Common issues and workarounds¶

If your build has failed, try one of the following hacky workarounds. If that doesn't work, try google.

(linux) C compiler doesn't see the `USER` variable¶

Add USER to the build/script-env list of variables to be inherited into build.sh:

build:
  script_env:
    - USER

(macOS) Build tries to use the wrong version of the macOS SDK¶

Make sure and read Installing the macOS SDK, specifically the lines relating to the ~/conda_build_config.yaml file.

Docker containers¶

If you prefer to work in a pristine environment, you can use the quay.io/condaforge/linux-anvil-comp7 docker container; this is what is used by conda-forge feedstock CIs when building packages for production.