Blocked by #59

tl;dr: its a mess but could work for now.

Look into skipping jobs that do not need to run. e.g.:

• build-base-debian does not need to run if there is no change to message-queue/container/base-debian/**
• build-base-debian-test does not need to run if there is no change to message-queue/container/base-debian/** and its dependent job (build-base-debian).
• etc ...

The issue is we will need to determine and propagate the sha from previous push to pull the previous build for the next job (e.g. build-producer-python).

Options:

• https://github.com/tj-actions/changed-files
• split into separate workflows to use the on: path: []. This would likely be most efficient but 1. doesn't feel right. 2. would require separate workflow for each job. 3. Unclear how this would work with skipped workflows. (https://stackoverflow.com/questions/58457140/dependencies-between-workflows-on-github-actions)
  • This could actually work as it would make sense to have a workflow for each of MQ producer separate as they are logically independent. We would also get a badge for each producer. Its just a question of connecting it on top of infra build workflow.
  • Ok, this sucks. There actually isn't a way to combine on: workflow and on: path.
  • Maybe using workflow_call for common ones? That's unlikely to work as they would either all run anyway or stop the pipeline because concurrency restrictions. Might still be worth even if they re-run the common jobs as those are relatively inexpensive.
  • This is not going to work because concurrency group queue is capped at 1. Queing more just cancels the pending job.
  • The obvious solution would be to have main workflow run the common jobs, then run the rest as called workflows. The issue is the called workflows can't use on: path and they won't give us badges either.

So that leaves 2 options:

• Run the common as separate workflow and wait for the images to appear in registry in other workflows.
  This would be relatively simple but fragile. Common workflow failing would result into other workflows timeouting (custom impl) and it would still consume GHA minutes (Question: are GHA minutes CPU clock or wall clock?). (e.g.: https://github.com/marketplace/actions/wait-on-check )

So realistically that leaves only one option:

• Run common in main workflow, run the rest using called workflows and custom path filtering (tj-actions) and custom badge generation.

• The jobs can't be skipped by an if as that would skip the dependers as well. Individual steps could be skipped. Maybe some logic that would set the old sha as output for the depender. Actually tj-actions/changed-files could provide the previous sha on push.

• Or maybe a custom action that would output exactly what needs to run next.

• Look into how badges work
  https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/monitoring-workflows/adding-a-workflow-status-badge
  badge = (workflow, branch, event)

• Look into how GHA behaves when multiple workflows define exact same job (is the job executed just once or twice for each workflow?)
  Yeah it just runs the job multiple times ( https://github.com/jan-matejka/test/tree/example/same-job )

• Looks like there is no way to define workflows outside /.github. Symlinks do not work and neither do called workflows from there (at leats relative ones).

Another things:

• Can't combine paths and paths-ignore.
• The current approach will no longer work when we will add docs building.