`python/test`¶

Configures jobs to test a Python project.

Description¶

This component installs the target Python project and then runs pytest to execute the project test suite.

In the simplest case (with no inputs given), this component configures a job matrix that, for multiple versions of Python, executes roughly the following commands:

python -m pip install pytest pytest-cov
python -m pip install .
python -m pytest --cov . --junit-xml junit.xml .

However, the various inputs, and the layout of the project, dynamically influence the script.

Usage¶

include:
  - component: git.ligo.org/computing/gitlab/components/python/test@<VERSION>

Inputs¶

Input	Default value	Description
`job_name`	`python_test`	The name to give the job.
`stage`	`test`	Pipeline stage to add jobs to
`python`	`"python"`	Name/path of Python interpreter to use
`conda`	`false`	If `true`, use `conda` to populate the test environment. See Conda below for more details
`needs`	`[]`	List of jobs whose artifacts are needed by the test jobs
`install_target`	`"."`	Path/file/package to install (supports wildcards)
`install_extra`	None	Name of the extra feature group(s) to install (comma-separated)
`pytest_options`	`""`	Extra options to pass to pytest
`extra_test_commands`	`[]`	Extra test commands to run (after pytest)
`coverage_rcfile`	None	Path to coverage.py configuration file (helps with providing accurate coverage measurements, autodiscovered in most cases)

Notes¶

Conda¶

If conda: true is given in the inputs to the python/test component, the following changes are made to the configured job matrix:

The job matrix is called { job_name }_conda, e.g. python_test_conda.
A custom before_script is configured that uses pip2conda to create a conda environment called test in which the tests will run.
The install_extra input can accept a new value "--all" which triggers pip2conda to install packages for all optional dependencies keys in the pyproject.toml/setup.cfg/setup.py files.
Jobs will cache downloaded conda packages as well as pip downloads.

The execution of pytest and the details of the coverage reporting are unchanged.

Automatic test and coverage reporting¶

The tests are executed using pytest and automatically include coverage reporting using pytest-cov and Unit test reports.

Coverage is combined in the `.post` pipeline stage¶

GitLab's default coverage parser averages coverage values for multiple jobs. This down-ranks coverage from projects that may have lower coverage on some platforms (e.g. Windows).

To get around this limitation, the python/test component configures a python_coverage job in the .post that uses coverage.py's combine command to correctly merge coverage reports from multiple jobs. This job then presents a single coverage statistic for the whole pipeline.

Customisation¶

pytest¶

pytest's behaviour can be be customised in one of the following ways (ordered by preference, most recommended to least):

by specifying the tool.pytest.ini_options table in the project pyproject.toml configuration file.
Configure pytest in pyproject.toml
```
[tool.pytest.ini_options]
addopts = "-ra --cov myproject"
```

give the pytest_options input

Configure pytest through the component inputs

include:
  - component: git.ligo.org/computing/gitlab/components/python/test@<VERSION>
    inputs:
      pytest_options: "-ra -v"

set the PYTEST_ADDOPTS variable for the python_test job:

Set PYTEST_ADDOPTS for the python_test job

include:
  - component: git.ligo.org/computing/gitlab/components/python/test@<VERSION>

python_test:
  variables:
    PYTEST_ADDOPTS: "-k 'not bad_test'

coverage.py¶

Coverage gathering and reporting should be customised via the [tool.coverage] section of the project pyproject.toml configuration file.

Configure coverage.py in pyproject.toml

[tool.coverage.paths]
# map paths from installed locations back to project source
source = [
  "myproject/",  # <-- source path
  "*/myproject/",  # <-- any installed path
]

[tool.coverage.report]
precision = 1
# omit auto-generated version file
omit = [
  "*/_version.py",
]

Examples¶

Testing a Python project¶

Run tests for a Python project

include:
  - component: git.ligo.org/computing/gitlab/components/python/test@<VERSION>
    inputs:
      install_extra: "test"
      pytest_options: "-ra -v"
      python_versions:
        - "3.10"
        - "3.11"
        - "3.12"

Executing tests from an installed library¶

To run tests for an installed library, rather than from the project directory, specify install_target to install the distribution and pytest_options to target it using the --pyargs option:

Running tests for an installed library

include:
  - component: git.ligo.org/computing/gitlab/components/python/wheel@<VERSION>
    inputs:
      project_dir: example/
  - component: git.ligo.org/computing/gitlab/components/python/test@<VERSION>
    inputs:
      job_name: "pytest"
      install_target: "*.whl"
      install_extra: "test"
      needs: [wheel]
      pytest_options: "-ra --pyargs example_project"
      extra_test_commands:
        - my_cli --help
      python_versions:
        - "3.10"
        - "3.11"
        - "3.12"

python/test¶