Component: solutions, auth, ingress

Why this is needed:

Today, we expect Solution Operators to handle deploying their UI (as this feature does not need to be mandatory in anyway, nor does a Solution UI need to be an "admin UI", i.e. exposed on the control-plane and using Dex as an OIDC provider).
However, we cannot reasonably expect such Operators to entirely manage the configuration of MetalK8s specifics (namely the Ingress object with the right path, depending on the environment, and the Dex static client it requires).

What should be done:

We need to define some automation around this. Since the UI is not deployed by our scripts (we only deploy the Operator), we cannot know in advance when/if it will be. As such, it would be a good idea to implement our own controller watching K8s API for changes, reacting by applying the required modifications as Admin UIs are added / removed.

Implementation proposal (strongly recommended):

The controller can be implemented using Salt Beacons + Reactors for now, if we really want to, or as a separate container using the Go library controller-runtime, akin to standard K8s controllers / operators.

• it would watch Service objects with the solutions.metalk8s.scality.com/admin-ui='' label
• using annotations on such Services, it would generate the right Ingress object (control-plane class, path based on the Service's namespace solutions.metalk8s.scality.com/environment label value)
• assuming the Service requests it (through other annotations), it would generate a static OIDC client for Dex (a unique ID + secret, and a computed redirect URI) and apply it with Salt (or using any other mean we have for configuring Dex in a persistent manner)
• it would edit/create a ConfigMap mounted by the UI container, adding:
  • a key for theming options, based on the cluster-wide options by default, potentially with an annotation on the Service to opt-out of this behaviour in case one wants to only style a single UI (e.g. for white labeling)
  • a key for the OIDC client configuration, namely the provider's URL, the client ID and secret

Test plan:

Validate all this using the example-solution.