Skip to content

CI: K8sVerifier Runs the kernel verifier against Cilium's BPF datapath: terminating containers are not deleted after timeout #18895

Closed
@maintainer-s-little-helper

Description

Test Name

K8sVerifier Runs the kernel verifier against Cilium's BPF datapath

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show.
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-net-next/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0019014c0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-896gh echo-694c58bbf4-fr4ck]",
        err: <*errors.errorString | 0xc002911860>{
            s: "Pods are still terminating: [echo-694c58bbf4-896gh echo-694c58bbf4-fr4ck]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-net-next/src/github.com/cilium/cilium/vendor/github.com/onsi/ginkgo/internal/leafnodes/runner.go:64

Standard Output

Click to show.


Standard Error

Click to show.
07:46:20 STEP: Running BeforeAll block for EntireTestsuite K8sVerifier
07:46:20 STEP: Ensuring the namespace kube-system exists
07:46:20 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
07:46:20 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
07:46:20 STEP: Deleting pods [echo-694c58bbf4-896gh,echo-694c58bbf4-fr4ck] in namespace default
07:46:20 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-896gh,echo-694c58bbf4-fr4ck)
07:46:20 STEP: Deleting cep [echo-694c58bbf4-896gh,echo-694c58bbf4-fr4ck] in namespace default
07:46:20 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-896gh,echo-694c58bbf4-fr4ck)
07:46:30 STEP: Unable to delete pods echo-694c58bbf4-896gh with 'kubectl -n default delete pods echo-694c58bbf4-896gh': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-896gh" deleted
	 
Stderr:
 	 

07:46:30 STEP: Unable to delete pods echo-694c58bbf4-fr4ck with 'kubectl -n default delete pods echo-694c58bbf4-fr4ck': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-fr4ck" deleted
	 
Stderr:
 	 

07:46:30 STEP: WaitForSinglePod(namespace="default", podname="test-verifier")
07:46:32 STEP: waitForSinglePod(namespace="default", podname="test-verifier", minRequired=1) => <nil>
07:46:48 STEP: Cleaning bpf_lxc build files
07:46:48 STEP: Building bpf_lxc object file
07:46:52 STEP: Running the verifier test script with bpf_lxc
07:46:54 STEP: Cleaning bpf_lxc build files
07:46:54 STEP: Building bpf_lxc object file
07:46:56 STEP: Running the verifier test script with bpf_lxc
07:46:57 STEP: Cleaning bpf_lxc build files
07:46:57 STEP: Building bpf_lxc object file
07:46:59 STEP: Running the verifier test script with bpf_lxc
07:46:59 STEP: Cleaning bpf_host build files
07:47:00 STEP: Building bpf_host object file
07:47:03 STEP: Running the verifier test script with bpf_host
07:47:05 STEP: Cleaning bpf_host build files
07:47:05 STEP: Building bpf_host object file
07:47:07 STEP: Running the verifier test script with bpf_host
07:47:09 STEP: Cleaning bpf_host build files
07:47:09 STEP: Building bpf_host object file
07:47:11 STEP: Running the verifier test script with bpf_host
07:47:12 STEP: Cleaning bpf_xdp build files
07:47:12 STEP: Building bpf_xdp object file
07:47:14 STEP: Running the verifier test script with bpf_xdp
07:47:15 STEP: Cleaning bpf_overlay build files
07:47:15 STEP: Building bpf_overlay object file
07:47:18 STEP: Running the verifier test script with bpf_overlay
07:47:19 STEP: Cleaning bpf_sock build files
07:47:19 STEP: Building bpf_sock object file
07:47:20 STEP: Running the verifier test script with bpf_sock
=== Test Finished at 2022-02-22T07:47:20Z====
07:47:20 STEP: Running AfterEach for block EntireTestsuite

07:47:20 STEP: Running AfterAll block for EntireTestsuite K8sVerifier
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0019014c0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-896gh echo-694c58bbf4-fr4ck]",
        err: <*errors.errorString | 0xc002911860>{
            s: "Pods are still terminating: [echo-694c58bbf4-896gh echo-694c58bbf4-fr4ck]",
        },
    }
to be nil


ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next//551/artifact/test_results_Cilium-PR-K8s-1.23-kernel-net-next_551_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next/551/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Activity

joestringer

joestringer commented on Feb 23, 2022

@joestringer
Member

Similarities to #18634 , but observed in the tree after merging #18448. Needs more investigation of the specific logs here to understand which test isn't cleaning up after itself, then look into why that test doesn't delete the pods correctly before continuing.

brb

brb commented on Feb 23, 2022

@brb
Member

Just copy-pasting what I wrote internally regarding this:

I think that there might be a race between terminating pods and waiting with ExpectAllPodsTerminated(). The latter is checking whether there are any pods in the terminating state. What happens if the pod gets into terminating state after the latter has been executed? The waiting function would just return.

maintainer-s-little-helper

maintainer-s-little-helper commented on Feb 24, 2022

@maintainer-s-little-helper
Author

PR #18894 hit this flake with 93.60% similarity:

Click to show.

Test Name

K8sVerifier Runs the kernel verifier against Cilium's BPF datapath

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show.
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-net-next/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000c13c40>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-26qsc echo-694c58bbf4-tmmhp]",
        err: <*errors.errorString | 0xc0016ebc10>{
            s: "Pods are still terminating: [echo-694c58bbf4-26qsc echo-694c58bbf4-tmmhp]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-net-next/src/github.com/cilium/cilium/vendor/github.com/onsi/ginkgo/internal/leafnodes/runner.go:64

Standard Output

Click to show.


Standard Error

Click to show.
02:30:53 STEP: Running BeforeAll block for EntireTestsuite K8sVerifier
02:30:54 STEP: Ensuring the namespace kube-system exists
02:30:54 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
02:30:54 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
02:30:54 STEP: Deleting pods [echo-694c58bbf4-26qsc,echo-694c58bbf4-tmmhp] in namespace default
02:30:54 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-26qsc,echo-694c58bbf4-tmmhp)
02:30:54 STEP: Deleting cep [echo-694c58bbf4-26qsc,echo-694c58bbf4-tmmhp] in namespace default
02:30:54 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-26qsc,echo-694c58bbf4-tmmhp)
02:31:04 STEP: Unable to delete pods echo-694c58bbf4-26qsc with 'kubectl -n default delete pods echo-694c58bbf4-26qsc': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-26qsc" deleted
	 
Stderr:
 	 

02:31:04 STEP: Unable to delete pods echo-694c58bbf4-tmmhp with 'kubectl -n default delete pods echo-694c58bbf4-tmmhp': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-tmmhp" deleted
	 
Stderr:
 	 

02:31:04 STEP: WaitForSinglePod(namespace="default", podname="test-verifier")
02:31:07 STEP: waitForSinglePod(namespace="default", podname="test-verifier", minRequired=1) => <nil>
02:31:25 STEP: Cleaning bpf_lxc build files
02:31:25 STEP: Building bpf_lxc object file
02:31:30 STEP: Running the verifier test script with bpf_lxc
02:31:32 STEP: Cleaning bpf_lxc build files
02:31:32 STEP: Building bpf_lxc object file
02:31:34 STEP: Running the verifier test script with bpf_lxc
02:31:36 STEP: Cleaning bpf_lxc build files
02:31:37 STEP: Building bpf_lxc object file
02:31:38 STEP: Running the verifier test script with bpf_lxc
02:31:39 STEP: Cleaning bpf_host build files
02:31:39 STEP: Building bpf_host object file
02:31:43 STEP: Running the verifier test script with bpf_host
02:31:45 STEP: Cleaning bpf_host build files
02:31:45 STEP: Building bpf_host object file
02:31:47 STEP: Running the verifier test script with bpf_host
02:31:49 STEP: Cleaning bpf_host build files
02:31:49 STEP: Building bpf_host object file
02:31:51 STEP: Running the verifier test script with bpf_host
02:31:52 STEP: Cleaning bpf_xdp build files
02:31:52 STEP: Building bpf_xdp object file
02:31:55 STEP: Running the verifier test script with bpf_xdp
02:31:56 STEP: Cleaning bpf_overlay build files
02:31:56 STEP: Building bpf_overlay object file
02:31:58 STEP: Running the verifier test script with bpf_overlay
02:31:59 STEP: Cleaning bpf_sock build files
02:31:59 STEP: Building bpf_sock object file
02:32:00 STEP: Running the verifier test script with bpf_sock
=== Test Finished at 2022-02-24T02:32:01Z====
02:32:01 STEP: Running AfterEach for block EntireTestsuite

02:32:01 STEP: Running AfterAll block for EntireTestsuite K8sVerifier
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000c13c40>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-26qsc echo-694c58bbf4-tmmhp]",
        err: <*errors.errorString | 0xc0016ebc10>{
            s: "Pods are still terminating: [echo-694c58bbf4-26qsc echo-694c58bbf4-tmmhp]",
        },
    }
to be nil


ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next//561/artifact/1a1c6d78_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next//561/artifact/test_results_Cilium-PR-K8s-1.23-kernel-net-next_561_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next/561/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

maintainer-s-little-helper

maintainer-s-little-helper commented on Feb 25, 2022

@maintainer-s-little-helper
Author

PR #18915 hit this flake with 94.74% similarity:

Click to show.

Test Name

K8sVerifier Runs the kernel verifier against Cilium's BPF datapath

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show.
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-net-next/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000adaf80>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-h92sm echo-694c58bbf4-zb5sb]",
        err: <*errors.errorString | 0xc002567b60>{
            s: "Pods are still terminating: [echo-694c58bbf4-h92sm echo-694c58bbf4-zb5sb]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-net-next/src/github.com/cilium/cilium/vendor/github.com/onsi/ginkgo/internal/leafnodes/runner.go:64

Standard Output

Click to show.


Standard Error

Click to show.
02:50:19 STEP: Running BeforeAll block for EntireTestsuite K8sVerifier
02:50:19 STEP: Ensuring the namespace kube-system exists
02:50:19 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
02:50:19 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
02:50:19 STEP: Deleting cep [echo-694c58bbf4-h92sm,echo-694c58bbf4-zb5sb] in namespace default
02:50:19 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-h92sm,echo-694c58bbf4-zb5sb)
02:50:19 STEP: Deleting pods [echo-694c58bbf4-h92sm,echo-694c58bbf4-zb5sb] in namespace default
02:50:19 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-h92sm,echo-694c58bbf4-zb5sb)
02:50:29 STEP: Unable to delete pods echo-694c58bbf4-h92sm with 'kubectl -n default delete pods echo-694c58bbf4-h92sm': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-h92sm" deleted
	 
Stderr:
 	 

02:50:29 STEP: Unable to delete pods echo-694c58bbf4-zb5sb with 'kubectl -n default delete pods echo-694c58bbf4-zb5sb': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-zb5sb" deleted
	 
Stderr:
 	 

02:50:30 STEP: WaitForSinglePod(namespace="default", podname="test-verifier")
02:50:32 STEP: waitForSinglePod(namespace="default", podname="test-verifier", minRequired=1) => <nil>
02:50:50 STEP: Cleaning bpf_lxc build files
02:50:50 STEP: Building bpf_lxc object file
02:50:54 STEP: Running the verifier test script with bpf_lxc
02:50:56 STEP: Cleaning bpf_lxc build files
02:50:56 STEP: Building bpf_lxc object file
02:50:58 STEP: Running the verifier test script with bpf_lxc
02:50:59 STEP: Cleaning bpf_lxc build files
02:50:59 STEP: Building bpf_lxc object file
02:51:01 STEP: Running the verifier test script with bpf_lxc
02:51:02 STEP: Cleaning bpf_host build files
02:51:02 STEP: Building bpf_host object file
02:51:06 STEP: Running the verifier test script with bpf_host
02:51:08 STEP: Cleaning bpf_host build files
02:51:08 STEP: Building bpf_host object file
02:51:10 STEP: Running the verifier test script with bpf_host
02:51:12 STEP: Cleaning bpf_host build files
02:51:12 STEP: Building bpf_host object file
02:51:14 STEP: Running the verifier test script with bpf_host
02:51:15 STEP: Cleaning bpf_xdp build files
02:51:15 STEP: Building bpf_xdp object file
02:51:18 STEP: Running the verifier test script with bpf_xdp
02:51:19 STEP: Cleaning bpf_overlay build files
02:51:19 STEP: Building bpf_overlay object file
02:51:21 STEP: Running the verifier test script with bpf_overlay
02:51:22 STEP: Cleaning bpf_sock build files
02:51:22 STEP: Building bpf_sock object file
02:51:23 STEP: Running the verifier test script with bpf_sock
=== Test Finished at 2022-02-25T02:51:24Z====
02:51:24 STEP: Running AfterEach for block EntireTestsuite

02:51:24 STEP: Running AfterAll block for EntireTestsuite K8sVerifier
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000adaf80>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-h92sm echo-694c58bbf4-zb5sb]",
        err: <*errors.errorString | 0xc002567b60>{
            s: "Pods are still terminating: [echo-694c58bbf4-h92sm echo-694c58bbf4-zb5sb]",
        },
    }
to be nil


ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next//571/artifact/test_results_Cilium-PR-K8s-1.23-kernel-net-next_571_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next/571/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

maintainer-s-little-helper

maintainer-s-little-helper commented on Mar 3, 2022

@maintainer-s-little-helper
Author

PR #19019 hit this flake with 92.75% similarity:

Click to show.

Test Name

K8sVerifier Runs the kernel verifier against Cilium's BPF datapath

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show.
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-net-next/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc001313280>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-494rj echo-694c58bbf4-mhx4l]",
        err: <*errors.errorString | 0xc0009cb640>{
            s: "Pods are still terminating: [echo-694c58bbf4-494rj echo-694c58bbf4-mhx4l]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-net-next/src/github.com/cilium/cilium/vendor/github.com/onsi/ginkgo/internal/leafnodes/runner.go:64

Standard Output

Click to show.


Standard Error

Click to show.
15:10:50 STEP: Running BeforeAll block for EntireTestsuite K8sVerifier
15:10:50 STEP: Ensuring the namespace kube-system exists
15:10:50 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
15:10:51 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
15:10:51 STEP: Deleting cep [echo-694c58bbf4-494rj,echo-694c58bbf4-mhx4l] in namespace default
15:10:51 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-494rj,echo-694c58bbf4-mhx4l)
15:10:51 STEP: Deleting pods [echo-694c58bbf4-494rj,echo-694c58bbf4-mhx4l] in namespace default
15:10:51 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-494rj,echo-694c58bbf4-mhx4l)
15:11:01 STEP: Unable to delete pods echo-694c58bbf4-mhx4l with 'kubectl -n default delete pods echo-694c58bbf4-mhx4l': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-mhx4l" deleted
	 
Stderr:
 	 

15:11:01 STEP: Unable to delete pods echo-694c58bbf4-494rj with 'kubectl -n default delete pods echo-694c58bbf4-494rj': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-494rj" deleted
	 
Stderr:
 	 

15:11:01 STEP: WaitForSinglePod(namespace="default", podname="test-verifier")
15:11:03 STEP: waitForSinglePod(namespace="default", podname="test-verifier", minRequired=1) => <nil>
15:11:19 STEP: Cleaning bpf_lxc build files
15:11:20 STEP: Building bpf_lxc object file
15:11:24 STEP: Running the verifier test script with bpf_lxc
15:11:26 STEP: Cleaning bpf_lxc build files
15:11:26 STEP: Building bpf_lxc object file
15:11:28 STEP: Running the verifier test script with bpf_lxc
15:11:29 STEP: Cleaning bpf_lxc build files
15:11:29 STEP: Building bpf_lxc object file
15:11:31 STEP: Running the verifier test script with bpf_lxc
15:11:32 STEP: Cleaning bpf_host build files
15:11:32 STEP: Building bpf_host object file
15:11:36 STEP: Running the verifier test script with bpf_host
15:11:38 STEP: Cleaning bpf_host build files
15:11:38 STEP: Building bpf_host object file
15:11:40 STEP: Running the verifier test script with bpf_host
15:11:42 STEP: Cleaning bpf_host build files
15:11:42 STEP: Building bpf_host object file
15:11:44 STEP: Running the verifier test script with bpf_host
15:11:45 STEP: Cleaning bpf_xdp build files
15:11:45 STEP: Building bpf_xdp object file
15:11:48 STEP: Running the verifier test script with bpf_xdp
15:11:48 STEP: Cleaning bpf_overlay build files
15:11:49 STEP: Building bpf_overlay object file
15:11:51 STEP: Running the verifier test script with bpf_overlay
15:11:52 STEP: Cleaning bpf_sock build files
15:11:52 STEP: Building bpf_sock object file
15:11:53 STEP: Running the verifier test script with bpf_sock
=== Test Finished at 2022-03-03T15:11:54Z====
15:11:54 STEP: Running AfterEach for block EntireTestsuite

15:11:54 STEP: Running AfterAll block for EntireTestsuite K8sVerifier
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc001313280>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-494rj echo-694c58bbf4-mhx4l]",
        err: <*errors.errorString | 0xc0009cb640>{
            s: "Pods are still terminating: [echo-694c58bbf4-494rj echo-694c58bbf4-mhx4l]",
        },
    }
to be nil


ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next//622/artifact/test_results_Cilium-PR-K8s-1.23-kernel-net-next_622_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next/622/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

maintainer-s-little-helper

maintainer-s-little-helper commented on Mar 4, 2022

@maintainer-s-little-helper
Author

PR #18674 hit this flake with 94.74% similarity:

Click to show.

Test Name

K8sVerifier Runs the kernel verifier against Cilium's BPF datapath

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show.
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-net-next/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000c9dc80>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-8zwxk echo-694c58bbf4-zhjh9]",
        err: <*errors.errorString | 0xc0043c3800>{
            s: "Pods are still terminating: [echo-694c58bbf4-8zwxk echo-694c58bbf4-zhjh9]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-net-next/src/github.com/cilium/cilium/vendor/github.com/onsi/ginkgo/internal/leafnodes/runner.go:64

Standard Output

Click to show.


Standard Error

Click to show.
04:03:55 STEP: Running BeforeAll block for EntireTestsuite K8sVerifier
04:03:55 STEP: Ensuring the namespace kube-system exists
04:03:56 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
04:03:56 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
04:03:56 STEP: Deleting cep [echo-694c58bbf4-8zwxk,echo-694c58bbf4-zhjh9] in namespace default
04:03:56 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-8zwxk,echo-694c58bbf4-zhjh9)
04:03:56 STEP: Deleting pods [echo-694c58bbf4-8zwxk,echo-694c58bbf4-zhjh9] in namespace default
04:03:56 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-8zwxk,echo-694c58bbf4-zhjh9)
04:04:06 STEP: Unable to delete pods echo-694c58bbf4-zhjh9 with 'kubectl -n default delete pods echo-694c58bbf4-zhjh9': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-zhjh9" deleted
	 
Stderr:
 	 

04:04:06 STEP: Unable to delete pods echo-694c58bbf4-8zwxk with 'kubectl -n default delete pods echo-694c58bbf4-8zwxk': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-8zwxk" deleted
	 
Stderr:
 	 

04:04:26 STEP: WaitForSinglePod(namespace="default", podname="test-verifier")
04:04:28 STEP: waitForSinglePod(namespace="default", podname="test-verifier", minRequired=1) => <nil>
04:04:43 STEP: Cleaning bpf_lxc build files
04:04:43 STEP: Building bpf_lxc object file
04:04:47 STEP: Running the verifier test script with bpf_lxc
04:04:49 STEP: Cleaning bpf_lxc build files
04:04:49 STEP: Building bpf_lxc object file
04:04:51 STEP: Running the verifier test script with bpf_lxc
04:04:52 STEP: Cleaning bpf_lxc build files
04:04:52 STEP: Building bpf_lxc object file
04:04:54 STEP: Running the verifier test script with bpf_lxc
04:04:54 STEP: Cleaning bpf_host build files
04:04:54 STEP: Building bpf_host object file
04:04:58 STEP: Running the verifier test script with bpf_host
04:05:00 STEP: Cleaning bpf_host build files
04:05:00 STEP: Building bpf_host object file
04:05:02 STEP: Running the verifier test script with bpf_host
04:05:04 STEP: Cleaning bpf_host build files
04:05:04 STEP: Building bpf_host object file
04:05:06 STEP: Running the verifier test script with bpf_host
04:05:06 STEP: Cleaning bpf_xdp build files
04:05:07 STEP: Building bpf_xdp object file
04:05:09 STEP: Running the verifier test script with bpf_xdp
04:05:10 STEP: Cleaning bpf_overlay build files
04:05:10 STEP: Building bpf_overlay object file
04:05:13 STEP: Running the verifier test script with bpf_overlay
04:05:13 STEP: Cleaning bpf_sock build files
04:05:14 STEP: Building bpf_sock object file
04:05:14 STEP: Running the verifier test script with bpf_sock
=== Test Finished at 2022-03-04T04:05:15Z====
04:05:15 STEP: Running AfterEach for block EntireTestsuite

04:05:15 STEP: Running AfterAll block for EntireTestsuite K8sVerifier
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000c9dc80>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-8zwxk echo-694c58bbf4-zhjh9]",
        err: <*errors.errorString | 0xc0043c3800>{
            s: "Pods are still terminating: [echo-694c58bbf4-8zwxk echo-694c58bbf4-zhjh9]",
        },
    }
to be nil


ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next//627/artifact/test_results_Cilium-PR-K8s-1.23-kernel-net-next_627_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next/627/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

maintainer-s-little-helper

maintainer-s-little-helper commented on Mar 8, 2022

@maintainer-s-little-helper
Author

PR #18620 hit this flake with 93.89% similarity:

Click to show.

Test Name

K8sVerifier Runs the kernel verifier against Cilium's BPF datapath

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show.
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-net-next/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:427
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc001abb220>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-6sbkc echo-694c58bbf4-v456m]",
        err: <*errors.errorString | 0xc001dea430>{
            s: "Pods are still terminating: [echo-694c58bbf4-6sbkc echo-694c58bbf4-v456m]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-net-next/src/github.com/cilium/cilium/vendor/github.com/onsi/ginkgo/internal/leafnodes/runner.go:64

Standard Output

Click to show.


Standard Error

Click to show.
10:54:43 STEP: Running BeforeAll block for EntireTestsuite K8sVerifier
10:54:43 STEP: Ensuring the namespace kube-system exists
10:54:43 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
10:54:43 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
10:54:43 STEP: Deleting cep [echo-694c58bbf4-6sbkc,echo-694c58bbf4-v456m] in namespace default
10:54:43 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-6sbkc,echo-694c58bbf4-v456m)
10:54:43 STEP: Deleting pods [echo-694c58bbf4-6sbkc,echo-694c58bbf4-v456m] in namespace default
10:54:43 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-6sbkc,echo-694c58bbf4-v456m)
10:54:53 STEP: Unable to delete pods echo-694c58bbf4-v456m with 'kubectl -n default delete pods echo-694c58bbf4-v456m': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-v456m" deleted
	 
Stderr:
 	 

10:54:53 STEP: Unable to delete pods echo-694c58bbf4-6sbkc with 'kubectl -n default delete pods echo-694c58bbf4-6sbkc': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-6sbkc" deleted
	 
Stderr:
 	 

10:54:53 STEP: WaitForSinglePod(namespace="default", podname="test-verifier")
10:54:55 STEP: waitForSinglePod(namespace="default", podname="test-verifier", minRequired=1) => <nil>
10:55:11 STEP: Cleaning bpf_lxc build files
10:55:12 STEP: Building bpf_lxc object file
10:55:16 STEP: Running the verifier test script with bpf_lxc
10:55:18 STEP: Cleaning bpf_lxc build files
10:55:18 STEP: Building bpf_lxc object file
10:55:20 STEP: Running the verifier test script with bpf_lxc
10:55:21 STEP: Cleaning bpf_lxc build files
10:55:21 STEP: Building bpf_lxc object file
10:55:23 STEP: Running the verifier test script with bpf_lxc
10:55:24 STEP: Cleaning bpf_host build files
10:55:24 STEP: Building bpf_host object file
10:55:28 STEP: Running the verifier test script with bpf_host
10:55:30 STEP: Cleaning bpf_host build files
10:55:30 STEP: Building bpf_host object file
10:55:32 STEP: Running the verifier test script with bpf_host
10:55:34 STEP: Cleaning bpf_host build files
10:55:34 STEP: Building bpf_host object file
10:55:36 STEP: Running the verifier test script with bpf_host
10:55:36 STEP: Cleaning bpf_xdp build files
10:55:37 STEP: Building bpf_xdp object file
10:55:40 STEP: Running the verifier test script with bpf_xdp
10:55:41 STEP: Cleaning bpf_overlay build files
10:55:41 STEP: Building bpf_overlay object file
10:55:44 STEP: Running the verifier test script with bpf_overlay
10:55:45 STEP: Cleaning bpf_sock build files
10:55:45 STEP: Building bpf_sock object file
10:55:46 STEP: Running the verifier test script with bpf_sock
=== Test Finished at 2022-03-08T10:55:46Z====
10:55:46 STEP: Running AfterEach for block EntireTestsuite

10:55:46 STEP: Running AfterAll block for EntireTestsuite K8sVerifier
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc001abb220>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-6sbkc echo-694c58bbf4-v456m]",
        err: <*errors.errorString | 0xc001dea430>{
            s: "Pods are still terminating: [echo-694c58bbf4-6sbkc echo-694c58bbf4-v456m]",
        },
    }
to be nil


ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next//658/artifact/test_results_Cilium-PR-K8s-1.23-kernel-net-next_658_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-net-next/658/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

49 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Labels

ci/flakeThis is a known failure that occurs in the tree. Please investigate me!sig/datapathImpacts bpf/ or low-level forwarding details, including map management and monitor messages.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    CI: K8sVerifier Runs the kernel verifier against Cilium's BPF datapath: terminating containers are not deleted after timeout · Issue #18895 · cilium/cilium