Software change requests with the SCCB¶

Procedure enhancement proposals

To propose an enhancement to the procedure described below, see Procedure enhancements.

Overview of a request¶

Each request asks for approval to install/update a software package or configuration that can have an impact on users or ongoing analyses. In order that changes are minimally disruptive, the following procedure has been created to ensure that changes are transparent, and that their potential impact has been tested and is understood.

The basic procedure is defined by the status scoped label set

graph TD; New[Request opened]-->val[<code>status::validation</code> This change needs tested]; val-->requested[<code>status::requested</code> Change is tested and reviewed the SCCB now votes]; requested-->approved[<code>status::approved</code> SCCB approves, passes to Run Coordinator for deployment approval]; requested-->rejected[<code>status::rejected</code> The change request has been rejected]; approved-->deploy[<code>status::deploy</code> The change can now be deployed in production]; approved-->rejected; deploy-->deployed[<code>status::deployed</code> The change has been deployed, waiting for feedback]; rejected-->closed[Request closed]; deployed-->closed; New-->revreq[<code>review::required</code> A review statement is needed]; revreq-->revapp[<code>review::approved</code> The review team approves this change]; revapp-->requested;

Request procedure¶

Creating a request¶

Click here to open a new Request

Embrace the request template

Please ensure that you complete the template as fully as you can, to enable efficient handling of your request, including giving a description of the changes and their potential impact on downstream users.

The following labels will be added automatically to all newly opened requests:

The status::validation label

When to apply: as soon as a request is opened.
Who should apply: the SCCB.
What happens next: packaging, review, and testing.

The review::required label

When to apply: as soon as a request is opened.
Who should apply: the SCCB.
What happens next: the review team will be contacted about the status of the review.

Additionally, the necessary packaging request labels will be applied (e.g. rhel::requested)

Expedited requests¶

If you require expedited approval for immediate deployment (e.g. of a critical bug fix), please post a comment to that effect in the comments of the ticket.

The expedite label

When to apply: if this request needs to be approved and deployed immediately.
Who should apply: the requester.
What happens next: the SCCB will try and push a request through as fast as possibly, whilst still following each step of the procedure.

Validation¶

The status::validation stage of a request includes a few parallel tracks:

Packaging and distribution
Testing
Scientific review

Once these steps are all complete, the request moves to be considered formally by the SCCB.

The status::requested label

When to apply: once all review, packaging, and testing are complete.
Who should apply: the requester.
What happens next: the SCCB will vote.

Packaging and distribution¶

The software distribution for which the relevant software will be packaged and into which it will be distributed should be selected via the Distributions section of the request template.

For each distribution there is a set of labels that track the progress of a request for a change to that distribution, e.g:

Label	Description
`<distribution>::requested`	Requires packaging for this distribution
`<distribution>::packaged`	This package is available for this distribution
`<distribution>::testing`	This package is installed in a testing environment
`<distribution>::tested`	This package has been tested in a testing environment for this distribution
`<distribution>::staging`	This package is ready for inclusion in the production environment
`<distribution>::production`	This package is installed in the production environment

The <distribution>::<stage> distribution labels

When to apply: at the relevant stage in the packaging process.
Who should apply: a packaging team member.

The exception to this rule is the tested label (see below).

The procedure for packaging and distribution is dependent upon which distribution(s) is selected:

For each of the Conda, Debian, and Rocky Linux options, the package builds will be assisted by the IGWN Computing and Software group's packaging team, and deployment of the packages into the testing, staging (if appropriate), and production repositories/environments will be handled by that team.
For the GWCelery option, the necessary package(s) are expected to be available from PyPI and installable with pip.

The GWCelery team will coordinate inclusion of the requested package(s) in the appropriate testing ('playground') environment, and eventually in the production environment.
For the Other option, the requester is required to give details of how the software will be distributed into the relevant software environment, including details of how independent testing will be performed.

Testing¶

Once packages are available in a testing environment for the appropriate distribution, they must be tested thoroughly.

Automated testing (continuous integration)

Where possible, all software projects should employ automated testing, or continuous integration, to prevent harmful changes entering the codebase as far as possible. Demonstrating that your project has robust automated testing can speed up the testing time by reducing the amount of manual testing required.

The <distribution>::tested labels

When to apply: after a package has been evaluated in a testing environment for the relevant distribution.
Who should apply: the requester.

For testing, packages will be installed in the following locations:

Platform	Location(s)
Conda	The `igwn-testing` environment
Debian	NONE
GWCelery	The `emfollow-playground` environment
Rocky Linux 8	`ldas-pcdev4.ligo.caltech.edu`, `ldas-pcdev8.ligo.caltech.edu`
Rocky Linux 9	`ldas-pcdev15.ligo.caltech.edu`

Alternative distributions (including AWS, Kubernetes, virtualenv, etc.) should be installed into a canary testing environment as soon as packages/containers are available for the requested software.

Don't test in production

Production systems should not be used for testing under any circumstances.

Scientific review¶

All scientific software packages (e.g. those used in an analysis pipeline) must be approved by the relevant scientific review team before the SCCB will consider them for approval.

The review::approved label

When to apply: when the review team approves the requested change
Who should apply: a member of the review team.

Approval¶

SCCB¶

Once the status::requested label has been applied to a ticket, the SCCB membership will vote on whether to approve, or reject, the request.

For details of the SCCB group approval rules, please see §4.2 (Approval rules) of the SCCB Charge document LIGO-T1800406-v2.

The status::approved label

When to apply: when the SCCB approves a request.
Who should apply: the SCCB member providing the final approval.
What happens next: the run coordinator will consider the request.

Any member of the SCCB may apply the status::hold label to prevent automatic approval of the request (once it otherwise has enough votes). This should be applied by an SCCB member to address any issues with the request that were not identified during validation.

Run Coordinator¶

Once approved, if the request was made during an observing epoch, the request will be considered by the Run Coordinator(s). They may also post a comment on the ticket to indicate a preference as to how, or when, the new software should be deployed.

The status::deploy label

When to apply: when the Run Coordinator approves a request.
Who should apply: the lead Run Coordinator.
What happens next: the change will be deployed.

Deployment¶

When the status::deploy label is applied, the requested change can be deployed.

Unless special action is required, the following is the timeline for deployment of approved software, regardless of the deployment mechanism:

Scheduled deployment

If the request is approved by 12:00 Pacific Time (Los Angeles) on a Friday, it will be considered for deployment during the next available Tuesday maintenance slot.

Otherwise it will not be deployed and will be considered for a future maintenance period.

Exceptions can be discussed on the Monday JRPC Run-Coordination meeting.

Managed deployments will be coordinated with the CompSoft packaging team with the following labels:

conda::production
debian::production
rhel::production

Exceptional deployment

expedite requests will be considered for exceptional deployment at the next available opportunity. These will be scheduled on a case-by-case basis by the Run Coordinator.

The status::deployed label

When to apply: when the requested software has been deployed on the target production system
Who should apply: the individual who undertakes the deployment.
What happens next: the SCCB will wait for feedback before closing the ticket.

Closing a ticket¶

Tickets will be closed once the relevant software/configuration has been deployed onto the production system, or the request has been rejected.

After the status::deployed label has been applied, a nominal waiting period will be observed, allowing users to report issues with the newly deployed software, before the issue is closed.

Closing tickets

When to close: when production deployments have been deemed successfull.
Who should close: the SCCB

Useful information¶

Contact¶

If you need to contact any the SCCB directly about your request, either use the @sccb gitlab syntax in your message, or email sccb@ligo.org.

Labels¶

For a full overview of the many labels that may be applied to a request, see https://git.ligo.org/computing/sccb/-/labels?sort=name_asc.