GitLab-CI¶

What is continuous integration?¶

Continuous integration (CI) is the practice of running an automated pipeline of scripts to build and test a project after every change.

This allows maintainers to identify bugs early in the development cycle, ensuring that all code that is pushed into the main development branch is compliant with the requirements of the project.

If configured correctly (this is the default), CI pipelines will run for every merge request, meaning the modified code can be build and tested before changes are accepted into the repository.

The fork-branch-merge-request presented earlier can then be augmented to include CI at all stages

How to enable CI on GitLab¶

All Gitlab CI/CD is configured with a YAML-format file called .gitlab-ci.yml in the root of the project repository.

https://docs.gitlab.com/ee/ci/yaml/README.html

Examples¶

Running a simple script¶

A basic example could just check that your script runs without failing. Consider the example from https://git.ligo.org/computing/gitlab/examples/cicd-intro/:

image: python

test:
  script:
    - python hello_world.py

Here, a single job called test is configured to run python hello_world.py. If that command exits with code 0, the job passes, otherwise it fails.

An example of this pipeline running is here:

https://git.ligo.org/computing/gitlab/examples/cicd-intro/-/pipelines/620533

Building a (python) package¶

A more complicated example is available on the package branch of the same gitlab-example repository:

https://git.ligo.org/duncanmmacleod/gitlab-example/tree/package

Here the full YAML configuration is:

stages:
  - build
  - test

image: python

build:
  stage: build
  script:
    - python -m pip install build
    - python -m build --wheel --outdir .
  artifacts:
    paths:
      - "hello_world-*.whl"

test:
  stage: test
  parallel:
    matrix:
      - PYTHON_VERSION:
          - "3.11"
          - "3.12"
  image: python:${PYTHON_VERSION}
  needs:
    - build
  script:
    - python -m pip install hello_world-*.whl
    - hello-world

Here we define a more complicated pipeline with two stages:

graph TD;
    build-->py311["test: [3.11]"];
    build-->py312["test: [3.12]"];

The build job uses artifacts to store the output of its job (the built wheel) so that it can be used in later jobs.

This configuration also uses parallel:matrix to create multiple test jobs running in different images based on the PYTHON_VERSION variable.

An example of this pipeline running is here:

https://git.ligo.org/computing/gitlab/examples/cicd-intro-package/-/pipelines/620547

Other examples¶

gwdatafind/gwdatafind (python package with automated tests): : https://git.ligo.org/gwdatafind/gwdatafind/pipelines/61271

lscsoft/bayeswave (cmake build with binary packaging and tests): : https://git.ligo.org/lscsoft/bayeswave/pipelines/82380

lscsoft/lalsuite (multi-package, multi-distribution package suite with scheduled nightly jobs): : https://git.ligo.org/lscsoft/lalsuite/pipelines/81275

emfollow/gwcelery (python package with documentation, tests, and continuous deployment to multiple locations): : https://git.ligo.org/emfollow/gwcelery/pipelines/82277

Modular pipelines with CI/CD components¶

The above simple examples are easy to understand and maintain, however when developing complex applications, or working on multiple independent projects, maintaining high-quality CI/CD pipelines that track the latest best practices can be a daunting task.

To improve this situation, GitLab supports CI/CD components, reusable pipeline configuration units that create parts of the whole pipeline, and can be shared easily across multiple projects.

The IGWN Computing and Software Working Group maintain a suite of CI/CD components that are visible in the IGWN GitLab CI/CD Catalog:

https://git.ligo.org/explore/catalog/

For details, see Components.

GitLab Pages¶

GitLab supports building static web content with CI and hosting them through Gitlab with an extension called Pages.

An example of this can be seen for this webpage:

https://git.ligo.org/computing/conda

For more details, see

https://git.ligo.org/help/user/project/pages/index.html

Runners and tags¶

This section has been moved here.

Advanced concepts¶

Matlab¶

MATLAB can be used in Gitlab CI only from a runner machine that can access a license file or a network license manager.

Finding the License server information¶

The license information for the LIGO-Caltech MATLAB license can be discovered directly from any of the LDAS-CIT login hosts as follows:

cat /ldcg/matlab_r2021b/licenses/network.lic | grep ^SERVER | awk -v RS='\r?\n' '{ print $4 "@" $2 }'

The output of this command should then be stored in a masked or protected CI/CD variable with the name MLM_LICENSE_FILE.

Configuring .gitlab-ci.yml¶

The YAML configuration for a MATLAB job would then look something like this:

myjob:
  image: mathworks/matlab:r2021b
  tags:
    - cit
  script:
    - matlab -batch test

For more information on the MATLAB docker images, see here.

Multi-project pipelines¶

GitLab CI allows specifying triggers in one project that will launch a CI/CD pipeline in a downstream project. For example:

test:
  stage: test
  script: make check

downstream:
  stage: deploy
  trigger: my/otherproject

In this example, after the test job succeeds in the test stage, the downstream job starts. GitLab then creates a downstream pipeline in the my/otherproject project.

This is used in Computing and Software to rebuild downstream Docker containers if the base container is rebuilt, see here.

See Multi-project pipelines in the official GitLab Docs for full details.