Problem/Motivation

In #3414391: Define extra rule to jobs that have "needs: composer" we've introduced consistent behaviour for all the possible combinations of OPT_IN_* and SKIP_* variables in GitLab CI configurations of a project.

It is very likely that even more combinations will be introduced as new features get added to the templates.

This makes it impossible (if it isn't already) to test all those combinations and make sure that they ain't broken by any of the changes to the template files.

Proposed resolution

We can fix that by adding all the tests to the CI of this template project. Here is how that could look like:

  • We build a reference project that contains several branches, each of which with files present or absent, which are relevant for certain jobs to be run or not, e.g. *.js files for ESLINT or mkdocs.yml for PAGES
  • The CI configuration of that reference project gets a job in the .pre stage which can only be triggered manually. That will allow us to create pipelines that can be analysed but actually won't be executed - to save resources
  • Then we define a matrix of variable combinations that we want to test, together with a list of expected jobs, that should be created in the respective pipeline
  • The test job will then loop through that list and create a pipeline for each, together with the necessary variables. We can use the glab CLI tool for this and that will create pipelines, but they won't execute
  • With the same tool, we can receive the pipeline details which should contain a list of contained jobs. This can be compared to the list of expected jobs, and if that doesn't match, the pipeline fails and we get notified that something is broken

This approach is fully automated and easy to extend, since only the matrix needs to be extended when new features get added to the templates.

It should be considered whether that test should be executed by default or only under certain circumstances.

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

jurgenhaas created an issue. See original summary.

jonathan1055’s picture

This is a great idea and it would be really helpful to test lots of combinations. But we probably can't test all of them, because we have 8 SKIP_ variables and 6 OPT_IN_ variables, so the full matrix is 2 ** 14 = 16,384. But a great many combinations will be meaningless, so I am sure we can get a good set to test with. [I don't want to be negative about the idea at all]

jurgenhaas’s picture

You're absolutely right @jonathan1055, the matrix will be handcrafted. This is necessary also because we need to define the list of expected jobs for each test case in that matrix too.

fjgarlin’s picture

Yup, agree. A curated list of combinations should be enough.

fjgarlin’s picture

Title: Add tests for variable combinations of OPT_IN_* and SKIP_* » [PP-1] Add tests for variable combinations of OPT_IN_* and SKIP_*
Status: Active » Postponed
Related issues: +#3447071: Rename SKIP and OPT_IN variables to remove any implied default

We are planning on simplifying variable names here #3447071: Rename SKIP and OPT_IN variables to remove any implied default.
This should make things easier and more predictable. I don't know if this issue will be needed after but will postpone it for now.