OCPBUGS-44438: Remove non-matching feature-gated CVO manifests from payload #5093

petr-muller · 2024-11-08T18:27:15Z

What this PR does / why we need it:

CVO manifests contain some feature-gated ones:

since at least 4.16, there are feature-gated ClusterVersion CRDs
UpdateStatus feature is delivered through DevPreview (now) and TechPreview (later) feature set

We observed HyperShift CI jobs to fail when adding DevPreview-gated deployment manifests, which was unexpected. Investigating further, we discovered that HyperShift applies these manifests:

cluster-version-operator-665c5789d5-8sr59-bootstrap.log:

error: error parsing /var/payload/manifests/0000_00_update-status-controller_03_deployment-DevPreviewNoUpgrade.yaml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{".ReleaseImage":interface {}(nil)}

But even without these added manifests, this happens for existing ClusterVersion CRD manifests present in the payload:

$ ls -1 manifests/*clusterversions*crd.yaml
manifests/0000_00_cluster-version-operator_01_clusterversions-CustomNoUpgrade.crd.yaml
manifests/0000_00_cluster-version-operator_01_clusterversions-Default.crd.yaml
manifests/0000_00_cluster-version-operator_01_clusterversions-DevPreviewNoUpgrade.crd.yaml
manifests/0000_00_cluster-version-operator_01_clusterversions-TechPreviewNoUpgrade.crd.yaml

In a passing HyperShift CI job, the same log shows that all four manifests are applied instead of just one:

customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io created
customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured
customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured
customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured

This likely means that HyperShift hosted clusters end up using TechPreviewNoUpgrade ClusterVersion CRD?

The proper fix is probably to wire through the FeatureGate with desired featureset through CVOParams but we would need a bit of selection logic so for now we can just remove entropy by deleting all feature-gated manifests instead of stumbling at them.

Which issue(s) this PR fixes:

OCPBUGS-44438

Checklist

Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.

petr-muller · 2024-11-09T00:42:25Z

/retest

petr-muller · 2024-11-11T14:27:27Z

https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_hypershift/5093/pull-ci-openshift-hypershift-main-e2e-aks/1855048275799838720#1:build-log.txt%3A154-164

         --------------------------------------------------------------------------------
        RESPONSE 409: 409 Conflict
        ERROR CODE: StorageAccountAlreadyTaken
        --------------------------------------------------------------------------------
        {
          "error": {
            "code": "StorageAccountAlreadyTaken",
            "message": "The storage account named clusterpdlcb is already taken."
          }
        }
        --------------------------------------------------------------------------------

https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_hypershift/5093/pull-ci-openshift-hypershift-main-e2e-kubevirt-aws-ovn-reduced/1855048276131188736#1:build-log.txt%3A57

failed to acquire lease for "hypershift-quota-slice": status 502 Bad Gateway, status code 502

/test e2e-aks e2e-kubevirt-aws-ovn-reduced

openshift-ci-robot · 2024-11-11T15:35:06Z

@petr-muller: This pull request references Jira Issue OCPBUGS-44438, which is invalid:

expected the bug to target the "4.18.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

What this PR does / why we need it:

CVO manifests contain some feature-gated ones:

since at least 4.16, there are feature-gated ClusterVersion CRDs

UpdateStatus feature is delivered through DevPreview (now) and TechPreview (later) feature set

We observed HyperShift CI jobs to fail when adding DevPreview-gated deployment manifests, which was unexpected. Investigating further, we discovered that HyperShift applies these manifests:

cluster-version-operator-665c5789d5-8sr59-bootstrap.log:
error: error parsing /var/payload/manifests/0000_00_update-status-controller_03_deployment-DevPreviewNoUpgrade.yaml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{".ReleaseImage":interface {}(nil)}
But even without these added manifests, this happens for existing ClusterVersion CRD manifests present in the payload:
$ ls -1 manifests/*clusterversions*crd.yaml
manifests/0000_00_cluster-version-operator_01_clusterversions-CustomNoUpgrade.crd.yaml
manifests/0000_00_cluster-version-operator_01_clusterversions-Default.crd.yaml
manifests/0000_00_cluster-version-operator_01_clusterversions-DevPreviewNoUpgrade.crd.yaml
manifests/0000_00_cluster-version-operator_01_clusterversions-TechPreviewNoUpgrade.crd.yaml
In a passing HyperShift CI job, the same log shows that all four manifests are applied instead of just one:
customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io created
customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured
customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured
customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured
This likely means that HyperShift hosted clusters end up using TechPreviewNoUpgrade ClusterVersion CRD?

The proper fix is probably to wire through the FeatureGate with desired featureset through CVOParams but we would need a bit of selection logic so for now we can just remove entropy by deleting all feature-gated manifests instead of stumbling at them.

Which issue(s) this PR fixes:

I can file an OCPBUGS if this PR looks sane

Checklist

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

petr-muller · 2024-11-11T19:18:28Z

/hold

Holding until branch cut

petr-muller · 2024-11-26T17:12:17Z

/hold cancel

petr-muller · 2024-11-26T17:16:09Z

I have adopted Seth's approach from #5096. It is closer to desired behavior than my original proposal, I just tweaked it so it removes all non-matching manifests, instead of relying on a Default always present and the feature set one applied over it (which would not happen for CustomNoUpgrade).

This means that CVO will need to always provide a manifest for a given featureset if a certain resource needs to be applied, it cannot rely on Default one. Fortunately this is exactly the convention that even generated CRD follows. CVO manifests will need to either provide a non-gated manifest (without a Default/TechPreviewNoUpdate/... which will always be applied, or to provide a gated manifest for each featureset where the resource needs to be present.

petr-muller · 2024-11-26T17:16:54Z

/jira refresh

openshift-ci-robot · 2024-11-26T17:17:03Z

@petr-muller: This pull request references Jira Issue OCPBUGS-44438, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.19.0) matches configured target version for branch (4.19.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (jiezhao@redhat.com), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

muraee · 2024-11-26T17:28:11Z

please also add those changes here:

hypershift/control-plane-operator/controllers/hostedcontrolplane/v2/cvo/deployment.go

Line 124 in da24e17

    
           func preparePayloadScript(platformType hyperv1.PlatformType, oauthEnabled bool) string {

openshift-ci · 2024-12-04T15:27:23Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csrwng, petr-muller

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [csrwng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sjenning · 2024-12-04T15:28:12Z

/retest-required

openshift-ci-robot · 2024-12-04T16:18:58Z

/retest-required

Remaining retests: 0 against base HEAD 857ccab and 2 for PR HEAD d075673 in total

petr-muller · 2024-12-04T19:34:52Z

/retest-required

CVO manifests contain some feature-gated ones: - since at least 4.16, there are feature-gated `ClusterVersion` CRDs - `UpdateStatus` feature is delivered through DevPreview (now) and TechPreview (later) feature set We observed HyperShift CI jobs to fail when adding DevPreview-gated deployment manifests, which was unexpected. Investigating further, we discovered that HyperShift applies these manifests: [cluster-version-operator-665c5789d5-8sr59-bootstrap.log](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-version-operator/1091/pull-ci-openshift-cluster-version-operator-master-e2e-hypershift-conformance/1853764751801192448/artifacts/e2e-hypershift-conformance/dump/artifacts/namespaces/clusters-c01d0e18fc19f1e0757b/core/pods/logs/cluster-version-operator-665c5789d5-8sr59-bootstrap.log): ``` error: error parsing /var/payload/manifests/0000_00_update-status-controller_03_deployment-DevPreviewNoUpgrade.yaml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{".ReleaseImage":interface {}(nil)} ``` But even without these added manifests, this happens for existing `ClusterVersion` CRD manifests present in the payload: ```console ls -1 manifests/*clusterversions*crd.yaml  manifests/0000_00_cluster-version-operator_01_clusterversions-CustomNoUpgrade.crd.yaml  manifests/0000_00_cluster-version-operator_01_clusterversions-Default.crd.yaml  manifests/0000_00_cluster-version-operator_01_clusterversions-DevPreviewNoUpgrade.crd.yaml  manifests/0000_00_cluster-version-operator_01_clusterversions-TechPreviewNoUpgrade.crd.yaml ``` In a passing HyperShift CI job, the [same log](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-version-operator/1105/pull-ci-openshift-cluster-version-operator-master-e2e-hypershift-conformance/1854708481941049344/artifacts/e2e-hypershift-conformance/dump/artifacts/namespaces/clusters-a5d1b5c3fcb2445935f2/core/pods/logs/cluster-version-operator-96cdfbf7c-cxt9r-bootstrap.log) shows that all four manifests are applied instead of just one: ``` customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io created customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured ``` This likely means that HyperShift hosted clusters end up using TechPreviewNoUpgrade `ClusterVersion` CRD? Fix the problem by deleting all manifests whose filename indicates that it is a part of a featureset not matching the desired one. Co-authored-by: Seth Jennings <sjenning@redhat.com> Co-authored-by: Petr Muller <muller@redhat.com>

The fixtures are exact copies of the original, default featureset ones. This is to allow adding the new test separately so it can update the default fixtures and produce a clear difference from the defaults.

Fixtures updated with: ``` $ UPDATE=true go test ./control-plane-operator/controllers/hostedcontrolplane/... ./control-plane-operator/controllers/hostedcontrolplane/... ```

petr-muller · 2024-12-04T22:58:38Z

Had to rebase to pull fixture changes from https://github.com/openshift/hypershift/pull/5160/files#diff-33e09167c08ede07364a11e60abb341ce6e4141143fb1a7033c8af966f2c32aa to the TechPreview ones

petr-muller · 2024-12-05T11:16:52Z

/retest-required

openshift-ci · 2024-12-05T17:21:18Z

@petr-muller: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test e2e-aws

/test e2e-aws-4-18

/test e2e-aws-upgrade-hypershift-operator

/test e2e-kubevirt-aws-ovn-reduced

/test images

/test security

/test unit

/test verify

The following commands are available to trigger optional jobs:

/test e2e-aks

/test e2e-aws-metrics

/test e2e-azure-aks-ovn-conformance

/test e2e-conformance

/test e2e-kubevirt-aws-ovn

/test e2e-kubevirt-azure-ovn

/test e2e-kubevirt-metal-conformance

/test e2e-openstack

/test e2e-openstack-conformance

/test e2e-openstack-csi-cinder

/test e2e-openstack-csi-manila

/test e2e-openstack-nfv

/test okd-scos-e2e-aws-ovn

/test okd-scos-images

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-hypershift-main-e2e-aws

pull-ci-openshift-hypershift-main-e2e-aws-upgrade-hypershift-operator

pull-ci-openshift-hypershift-main-e2e-kubevirt-aws-ovn-reduced

pull-ci-openshift-hypershift-main-images

pull-ci-openshift-hypershift-main-okd-scos-e2e-aws-ovn

pull-ci-openshift-hypershift-main-security

pull-ci-openshift-hypershift-main-unit

pull-ci-openshift-hypershift-main-verify

In response to this:

/retest required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

petr-muller · 2024-12-05T17:21:20Z

/retest-required

petr-muller · 2024-12-06T15:19:39Z

/retest-required

petr-muller · 2024-12-09T15:41:44Z

/test all

petr-muller · 2024-12-09T20:09:40Z

/test e2e-aws-upgrade-hypershift-operator

Failed just on destroy 🤞

sjenning · 2024-12-10T14:34:01Z

/lgtm

petr-muller · 2024-12-11T17:27:54Z

/retest

petr-muller · 2024-12-11T18:00:52Z

LOL I was not expecting Konflux to piggyback on Prow commands

openshift-ci-robot · 2024-12-12T00:19:56Z

/retest-required

Remaining retests: 0 against base HEAD 07b4f6f and 2 for PR HEAD 8374ac1 in total

openshift-ci-robot · 2024-12-12T04:24:00Z

/retest-required

Remaining retests: 0 against base HEAD 1c35133 and 2 for PR HEAD 8374ac1 in total

petr-muller · 2024-12-12T09:06:51Z

/retest-required

openshift-ci · 2024-12-12T11:18:19Z

@petr-muller: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aks	`d075673`	link	false	`/test e2e-aks`
ci/prow/okd-scos-e2e-aws-ovn	`8374ac1`	link	false	`/test okd-scos-e2e-aws-ovn`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot · 2024-12-12T11:21:21Z

@petr-muller: Jira Issue OCPBUGS-44438: Some pull requests linked via external trackers have merged:

openshift/hypershift#5093

The following pull requests linked via external trackers have not merged:

openshift/hypershift#5096 is open

These pull request must merge or be unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-44438 has not been moved to the MODIFIED state.

In response to this:

What this PR does / why we need it:

CVO manifests contain some feature-gated ones:

since at least 4.16, there are feature-gated ClusterVersion CRDs

UpdateStatus feature is delivered through DevPreview (now) and TechPreview (later) feature set

We observed HyperShift CI jobs to fail when adding DevPreview-gated deployment manifests, which was unexpected. Investigating further, we discovered that HyperShift applies these manifests:

cluster-version-operator-665c5789d5-8sr59-bootstrap.log:
error: error parsing /var/payload/manifests/0000_00_update-status-controller_03_deployment-DevPreviewNoUpgrade.yaml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{".ReleaseImage":interface {}(nil)}
But even without these added manifests, this happens for existing ClusterVersion CRD manifests present in the payload:
$ ls -1 manifests/*clusterversions*crd.yaml
manifests/0000_00_cluster-version-operator_01_clusterversions-CustomNoUpgrade.crd.yaml
manifests/0000_00_cluster-version-operator_01_clusterversions-Default.crd.yaml
manifests/0000_00_cluster-version-operator_01_clusterversions-DevPreviewNoUpgrade.crd.yaml
manifests/0000_00_cluster-version-operator_01_clusterversions-TechPreviewNoUpgrade.crd.yaml
In a passing HyperShift CI job, the same log shows that all four manifests are applied instead of just one:
customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io created
customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured
customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured
customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured
This likely means that HyperShift hosted clusters end up using TechPreviewNoUpgrade ClusterVersion CRD?

The proper fix is probably to wire through the FeatureGate with desired featureset through CVOParams but we would need a bit of selection logic so for now we can just remove entropy by deleting all feature-gated manifests instead of stumbling at them.

Which issue(s) this PR fixes:

OCPBUGS-44438

Checklist

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

petr-muller · 2024-12-12T12:12:04Z

/jira refresh

openshift-ci-robot · 2024-12-12T12:12:10Z

@petr-muller: Jira Issue OCPBUGS-44438: All pull requests linked via external trackers have merged:

openshift/hypershift#5093

Jira Issue OCPBUGS-44438 has been moved to the MODIFIED state.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-bot · 2024-12-12T12:51:21Z

[ART PR BUILD NOTIFIER]

Distgit: hypershift
This PR has been included in build ose-hypershift-container-v4.19.0-202412121208.p0.gc5e01a9.assembly.stream.el9.
All builds following this will include this PR.

openshift-ci bot added the do-not-merge/needs-area label Nov 8, 2024

openshift-ci bot requested review from csrwng and hasueki November 8, 2024 18:27

openshift-ci bot added area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release and removed do-not-merge/needs-area labels Nov 8, 2024

petr-muller mentioned this pull request Nov 11, 2024

OTA-1269: USC: Add DevPreviewNoUpgrade-gated manifests openshift/cluster-version-operator#1107

Merged

2 tasks

petr-muller changed the title ~~Remove feature-gated CVO manifests from payload~~ OCPBUGS-44438: Remove feature-gated CVO manifests from payload Nov 11, 2024

sjenning mentioned this pull request Nov 11, 2024

WIP: OCPBUGS-44438: add featureset awareness of CVO manifests #5096

Closed

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 11, 2024

petr-muller force-pushed the devpreview-cvo branch from afe37bc to 2c5ab36 Compare November 26, 2024 17:12

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 26, 2024

petr-muller force-pushed the devpreview-cvo branch from 2c5ab36 to b04fab1 Compare November 26, 2024 17:12

petr-muller changed the title ~~OCPBUGS-44438: Remove feature-gated CVO manifests from payload~~ OCPBUGS-44438: Remove non-matching feature-gated CVO manifests from payload Nov 26, 2024

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Nov 26, 2024

petr-muller force-pushed the devpreview-cvo branch 3 times, most recently from f3a038d to 895e5be Compare November 28, 2024 19:36

petr-muller and others added 3 commits December 4, 2024 23:51

tests: populate TechPreviewNoUpgrade fixtures

51a85ca

The fixtures are exact copies of the original, default featureset ones. This is to allow adding the new test separately so it can update the default fixtures and produce a clear difference from the defaults.

TestControlPlaneComponents: test TechPreviewNoUpgrade featureSet

8374ac1

Fixtures updated with: ``` $ UPDATE=true go test ./control-plane-operator/controllers/hostedcontrolplane/... ./control-plane-operator/controllers/hostedcontrolplane/... ```

petr-muller force-pushed the devpreview-cvo branch from d075673 to 8374ac1 Compare December 4, 2024 22:54

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Dec 4, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 10, 2024

openshift-merge-bot bot merged commit c5e01a9 into openshift:main Dec 12, 2024
11 of 12 checks passed

petr-muller deleted the devpreview-cvo branch December 12, 2024 12:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCPBUGS-44438: Remove non-matching feature-gated CVO manifests from payload #5093

OCPBUGS-44438: Remove non-matching feature-gated CVO manifests from payload #5093

petr-muller commented Nov 8, 2024 •

edited

Loading

petr-muller commented Nov 9, 2024

petr-muller commented Nov 11, 2024

openshift-ci-robot commented Nov 11, 2024

petr-muller commented Nov 11, 2024

petr-muller commented Nov 26, 2024

petr-muller commented Nov 26, 2024 •

edited

Loading

petr-muller commented Nov 26, 2024

openshift-ci-robot commented Nov 26, 2024

muraee commented Nov 26, 2024

openshift-ci bot commented Dec 4, 2024

sjenning commented Dec 4, 2024

openshift-ci-robot commented Dec 4, 2024

petr-muller commented Dec 4, 2024

petr-muller commented Dec 4, 2024

petr-muller commented Dec 5, 2024

openshift-ci bot commented Dec 5, 2024

petr-muller commented Dec 5, 2024

petr-muller commented Dec 6, 2024

petr-muller commented Dec 9, 2024

petr-muller commented Dec 9, 2024

sjenning commented Dec 10, 2024

petr-muller commented Dec 11, 2024

petr-muller commented Dec 11, 2024

openshift-ci-robot commented Dec 12, 2024

openshift-ci-robot commented Dec 12, 2024

petr-muller commented Dec 12, 2024

openshift-ci bot commented Dec 12, 2024

openshift-ci-robot commented Dec 12, 2024

petr-muller commented Dec 12, 2024

openshift-ci-robot commented Dec 12, 2024

openshift-bot commented Dec 12, 2024

OCPBUGS-44438: Remove non-matching feature-gated CVO manifests from payload #5093

OCPBUGS-44438: Remove non-matching feature-gated CVO manifests from payload #5093

Conversation

petr-muller commented Nov 8, 2024 • edited Loading

petr-muller commented Nov 9, 2024

petr-muller commented Nov 11, 2024

openshift-ci-robot commented Nov 11, 2024

petr-muller commented Nov 11, 2024

petr-muller commented Nov 26, 2024

petr-muller commented Nov 26, 2024 • edited Loading

petr-muller commented Nov 26, 2024

openshift-ci-robot commented Nov 26, 2024

muraee commented Nov 26, 2024

openshift-ci bot commented Dec 4, 2024

sjenning commented Dec 4, 2024

openshift-ci-robot commented Dec 4, 2024

petr-muller commented Dec 4, 2024

petr-muller commented Dec 4, 2024

petr-muller commented Dec 5, 2024

openshift-ci bot commented Dec 5, 2024

petr-muller commented Dec 5, 2024

petr-muller commented Dec 6, 2024

petr-muller commented Dec 9, 2024

petr-muller commented Dec 9, 2024

sjenning commented Dec 10, 2024

petr-muller commented Dec 11, 2024

petr-muller commented Dec 11, 2024

openshift-ci-robot commented Dec 12, 2024

openshift-ci-robot commented Dec 12, 2024

petr-muller commented Dec 12, 2024

openshift-ci bot commented Dec 12, 2024

openshift-ci-robot commented Dec 12, 2024

petr-muller commented Dec 12, 2024

openshift-ci-robot commented Dec 12, 2024

openshift-bot commented Dec 12, 2024

petr-muller commented Nov 8, 2024 •

edited

Loading

petr-muller commented Nov 26, 2024 •

edited

Loading