Closed
Description
Test Name
K8sVerifier Runs the kernel verifier against Cilium's BPF datapath
Failure Output
FAIL: terminating containers are not deleted after timeout
Stacktrace
Click to show.
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-net-next/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
terminating containers are not deleted after timeout
Expected
<*fmt.wrapError | 0xc0019014c0>: {
msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-896gh echo-694c58bbf4-fr4ck]",
err: <*errors.errorString | 0xc002911860>{
s: "Pods are still terminating: [echo-694c58bbf4-896gh echo-694c58bbf4-fr4ck]",
},
}
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-net-next/src/github.com/cilium/cilium/vendor/github.com/onsi/ginkgo/internal/leafnodes/runner.go:64
Standard Output
Click to show.
Standard Error
Click to show.
07:46:20 STEP: Running BeforeAll block for EntireTestsuite K8sVerifier
07:46:20 STEP: Ensuring the namespace kube-system exists
07:46:20 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
07:46:20 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
07:46:20 STEP: Deleting pods [echo-694c58bbf4-896gh,echo-694c58bbf4-fr4ck] in namespace default
07:46:20 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-896gh,echo-694c58bbf4-fr4ck)
07:46:20 STEP: Deleting cep [echo-694c58bbf4-896gh,echo-694c58bbf4-fr4ck] in namespace default
07:46:20 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-896gh,echo-694c58bbf4-fr4ck)
07:46:30 STEP: Unable to delete pods echo-694c58bbf4-896gh with 'kubectl -n default delete pods echo-694c58bbf4-896gh': Exitcode: -1
Err: signal: killed
Stdout:
pod "echo-694c58bbf4-896gh" deleted
Stderr:
07:46:30 STEP: Unable to delete pods echo-694c58bbf4-fr4ck with 'kubectl -n default delete pods echo-694c58bbf4-fr4ck': Exitcode: -1
Err: signal: killed
Stdout:
pod "echo-694c58bbf4-fr4ck" deleted
Stderr:
07:46:30 STEP: WaitForSinglePod(namespace="default", podname="test-verifier")
07:46:32 STEP: waitForSinglePod(namespace="default", podname="test-verifier", minRequired=1) => <nil>
07:46:48 STEP: Cleaning bpf_lxc build files
07:46:48 STEP: Building bpf_lxc object file
07:46:52 STEP: Running the verifier test script with bpf_lxc
07:46:54 STEP: Cleaning bpf_lxc build files
07:46:54 STEP: Building bpf_lxc object file
07:46:56 STEP: Running the verifier test script with bpf_lxc
07:46:57 STEP: Cleaning bpf_lxc build files
07:46:57 STEP: Building bpf_lxc object file
07:46:59 STEP: Running the verifier test script with bpf_lxc
07:46:59 STEP: Cleaning bpf_host build files
07:47:00 STEP: Building bpf_host object file
07:47:03 STEP: Running the verifier test script with bpf_host
07:47:05 STEP: Cleaning bpf_host build files
07:47:05 STEP: Building bpf_host object file
07:47:07 STEP: Running the verifier test script with bpf_host
07:47:09 STEP: Cleaning bpf_host build files
07:47:09 STEP: Building bpf_host object file
07:47:11 STEP: Running the verifier test script with bpf_host
07:47:12 STEP: Cleaning bpf_xdp build files
07:47:12 STEP: Building bpf_xdp object file
07:47:14 STEP: Running the verifier test script with bpf_xdp
07:47:15 STEP: Cleaning bpf_overlay build files
07:47:15 STEP: Building bpf_overlay object file
07:47:18 STEP: Running the verifier test script with bpf_overlay
07:47:19 STEP: Cleaning bpf_sock build files
07:47:19 STEP: Building bpf_sock object file
07:47:20 STEP: Running the verifier test script with bpf_sock
=== Test Finished at 2022-02-22T07:47:20Z====
07:47:20 STEP: Running AfterEach for block EntireTestsuite
07:47:20 STEP: Running AfterAll block for EntireTestsuite K8sVerifier
FAIL: terminating containers are not deleted after timeout
Expected
<*fmt.wrapError | 0xc0019014c0>: {
msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-896gh echo-694c58bbf4-fr4ck]",
err: <*errors.errorString | 0xc002911860>{
s: "Pods are still terminating: [echo-694c58bbf4-896gh echo-694c58bbf4-fr4ck]",
},
}
to be nil
ZIP Links:
Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next/551/
If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.
Activity
joestringer commentedon Feb 23, 2022
Similarities to #18634 , but observed in the tree after merging #18448. Needs more investigation of the specific logs here to understand which test isn't cleaning up after itself, then look into why that test doesn't delete the pods correctly before continuing.
brb commentedon Feb 23, 2022
Just copy-pasting what I wrote internally regarding this:
maintainer-s-little-helper commentedon Feb 24, 2022
PR #18894 hit this flake with 93.60% similarity:
Click to show.
Test Name
Failure Output
Stacktrace
Click to show.
Standard Output
Click to show.
Standard Error
Click to show.
ZIP Links:
Click to show.
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next//561/artifact/1a1c6d78_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next//561/artifact/test_results_Cilium-PR-K8s-1.23-kernel-net-next_561_BDD-Test-PR.zip
Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next/561/
If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.
maintainer-s-little-helper commentedon Feb 25, 2022
PR #18915 hit this flake with 94.74% similarity:
Click to show.
Test Name
Failure Output
Stacktrace
Click to show.
Standard Output
Click to show.
Standard Error
Click to show.
ZIP Links:
Click to show.
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next//571/artifact/test_results_Cilium-PR-K8s-1.23-kernel-net-next_571_BDD-Test-PR.zip
Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next/571/
If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.
maintainer-s-little-helper commentedon Mar 3, 2022
PR #19019 hit this flake with 92.75% similarity:
Click to show.
Test Name
Failure Output
Stacktrace
Click to show.
Standard Output
Click to show.
Standard Error
Click to show.
ZIP Links:
Click to show.
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next//622/artifact/test_results_Cilium-PR-K8s-1.23-kernel-net-next_622_BDD-Test-PR.zip
Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next/622/
If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.
maintainer-s-little-helper commentedon Mar 4, 2022
PR #18674 hit this flake with 94.74% similarity:
Click to show.
Test Name
Failure Output
Stacktrace
Click to show.
Standard Output
Click to show.
Standard Error
Click to show.
ZIP Links:
Click to show.
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next//627/artifact/test_results_Cilium-PR-K8s-1.23-kernel-net-next_627_BDD-Test-PR.zip
Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next/627/
If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.
maintainer-s-little-helper commentedon Mar 8, 2022
PR #18620 hit this flake with 93.89% similarity:
Click to show.
Test Name
Failure Output
Stacktrace
Click to show.
Standard Output
Click to show.
Standard Error
Click to show.
ZIP Links:
Click to show.
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next//658/artifact/test_results_Cilium-PR-K8s-1.23-kernel-net-next_658_BDD-Test-PR.zip
Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next/658/
If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.
49 remaining items