Problem/Motivation
In #3414391: Define extra rule to jobs that have "needs: composer" we've introduced consistent behaviour for all the possible combinations of OPT_IN_*
and SKIP_*
variables in GitLab CI configurations of a project.
It is very likely that even more combinations will be introduced as new features get added to the templates.
This makes it impossible (if it isn't already) to test all those combinations and make sure that they ain't broken by any of the changes to the template files.
Proposed resolution
We can fix that by adding all the tests to the CI of this template project. Here is how that could look like:
- We build a reference project that contains several branches, each of which with files present or absent, which are relevant for certain jobs to be run or not, e.g.
*.js
files for ESLINT ormkdocs.yml
for PAGES - The CI configuration of that reference project gets a job in the
.pre
stage which can only be triggered manually. That will allow us to create pipelines that can be analysed but actually won't be executed - to save resources - Then we define a matrix of variable combinations that we want to test, together with a list of expected jobs, that should be created in the respective pipeline
- The test job will then loop through that list and create a pipeline for each, together with the necessary variables. We can use the
glab
CLI tool for this and that will create pipelines, but they won't execute - With the same tool, we can receive the pipeline details which should contain a list of contained jobs. This can be compared to the list of expected jobs, and if that doesn't match, the pipeline fails and we get notified that something is broken
This approach is fully automated and easy to extend, since only the matrix needs to be extended when new features get added to the templates.
It should be considered whether that test should be executed by default or only under certain circumstances.
Issue fork gitlab_templates-3416472
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
jonathan1055 CreditAttribution: jonathan1055 as a volunteer commentedThis is a great idea and it would be really helpful to test lots of combinations. But we probably can't test all of them, because we have 8
SKIP_
variables and 6OPT_IN_
variables, so the full matrix is 2 ** 14 = 16,384. But a great many combinations will be meaningless, so I am sure we can get a good set to test with. [I don't want to be negative about the idea at all]Comment #3
jurgenhaasYou're absolutely right @jonathan1055, the matrix will be handcrafted. This is necessary also because we need to define the list of expected jobs for each test case in that matrix too.
Comment #4
fjgarlin CreditAttribution: fjgarlin as a volunteer and at Drupal Association commentedYup, agree. A curated list of combinations should be enough.
Comment #6
fjgarlin CreditAttribution: fjgarlin as a volunteer and at Drupal Association commentedWe are planning on simplifying variable names here #3447071: Rename SKIP and OPT_IN variables to remove any implied default.
This should make things easier and more predictable. I don't know if this issue will be needed after but will postpone it for now.