Skip to content

PodDisruptionBudget with an empty selector {} being honored incorrectly #119731

Closed
@sairameshv

Description

What happened?

PodDisruptionBudget with an empty selector {} selects all the pods as expected.
But, during the drain /eviction, the pdb is not being honored correctly i.e. all the pods are getting evicted.

This issue is only observed with the empty selector and works as expected if a valid label selector is provided that selects all the pods in a namespace.

What did you expect to happen?

When an empty selector {} is provided in the PodDisruptionBudget's spec, the created PodDisruptionBudget needs to be honored by not evicting all the pods during the drain/eviction api call

How can we reproduce it (as minimally and precisely as possible)?

  • Create a simple busybox deployment with 3 replicas on a local cluster (or any cluster)
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: busybox
  name: busybox
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: busybox
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - image: busybox
        name: busybox
        command: ["sleep", "1000"]
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30

  • Create a pdb object with an empty selector {}
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: pdb
spec:
  minAvailable: 100%  
  selector: {}
  • Check the status of the created pdb object that has the expectedPods observed to be 3 (which is expected)
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  creationTimestamp: "2023-08-02T10:33:29Z"
  generation: 1
  name: pdb
  namespace: default
  resourceVersion: "495"
  uid: 448b956e-4d2f-4d25-b2c4-202d68d3061d
spec:
  minAvailable: 100%
  selector: {}
status:
  conditions:
  - lastTransitionTime: "2023-08-02T10:33:33Z"
    message: ""
    observedGeneration: 1
    reason: InsufficientPods
    status: "False"
    type: DisruptionAllowed
  currentHealthy: 3
  desiredHealthy: 3
  disruptionsAllowed: 0
  expectedPods: 3
  observedGeneration: 1
  • Call the drain api to see the pdb not being honored and all the pods being evicted
$kubectl drain 127.0.0.1 --delete-emptydir-data --dry-run=server

node/127.0.0.1 cordoned (server dry run)
evicting pod kube-system/coredns-6994cbdc59-rqs5s (server dry run)
evicting pod default/busybox-6b5dd5dcf8-2hmjf (server dry run)
evicting pod default/busybox-6b5dd5dcf8-x5dn6 (server dry run)
evicting pod default/busybox-6b5dd5dcf8-sgpdz (server dry run)
node/127.0.0.1 drained (server dry run)
  • Delete the above pdb object and create a new one with the label selector app: busybox
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: pdb
spec:
  minAvailable: 100%  
  selector: {}
[svanka@svanka pdb]$ cat pdb.yaml 
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: pdb-with-label-selector
spec:
  minAvailable: 100%  
  selector:
    matchLabels:
      app: busybox
  • Observe the expected status of the pdb object same as the old one with the expectedPods set to 3
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  creationTimestamp: "2023-08-02T10:38:00Z"
  generation: 1
  name: pdb-with-label-selector
  namespace: default
  resourceVersion: "579"
  uid: 5788a978-4a5c-40e3-9775-88e855abe38c
spec:
  minAvailable: 100%
  selector:
    matchLabels:
      app: busybox
status:
  conditions:
  - lastTransitionTime: "2023-08-02T10:38:00Z"
    message: ""
    observedGeneration: 1
    reason: InsufficientPods
    status: "False"
    type: DisruptionAllowed
  currentHealthy: 3
  desiredHealthy: 3
  disruptionsAllowed: 0
  expectedPods: 3
  observedGeneration: 1
  • Call the drain api to observe the pdb being honored as expected
$kubectl drain 127.0.0.1 --delete-emptydir-data --dry-run=server

node/127.0.0.1 cordoned (server dry run)
evicting pod kube-system/coredns-6994cbdc59-rqs5s (server dry run)
evicting pod default/busybox-6b5dd5dcf8-sgpdz (server dry run)
evicting pod default/busybox-6b5dd5dcf8-x5dn6 (server dry run)
evicting pod default/busybox-6b5dd5dcf8-2hmjf (server dry run)
error when evicting pods/"busybox-6b5dd5dcf8-2hmjf" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"busybox-6b5dd5dcf8-sgpdz" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"busybox-6b5dd5dcf8-x5dn6" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod default/busybox-6b5dd5dcf8-sgpdz (server dry run)
evicting pod default/busybox-6b5dd5dcf8-x5dn6 (server dry run)
evicting pod default/busybox-6b5dd5dcf8-2hmjf (server dry run)
error when evicting pods/"busybox-6b5dd5dcf8-sgpdz" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"busybox-6b5dd5dcf8-x5dn6" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"busybox-6b5dd5dcf8-2hmjf" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"4e136762ec0ea7216f04665eb983ca30633b1f9f", GitTreeState:"clean", BuildDate:"2023-06-10T04:19:41Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"28+", GitVersion:"v1.28.0-rc.0.46+3888b341f12c6e", GitCommit:"3888b341f12c6e7241428950394e61d8cc26d5f9", GitTreeState:"clean", BuildDate:"2023-08-02T11:02:24Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

local cluster - N/A

OS version

# On Linux:
$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.3 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.3"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.3"

$ uname -a
Linux svanka.remote.csb 4.18.0-193.28.1.el8_2.x86_64 #1 SMP Fri Oct 16 13:38:49 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

CRI-O

Related plugins (CNI, CSI, ...) and versions (if applicable)

Metadata

Assignees

Labels

kind/bugCategorizes issue or PR as related to a bug.needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.sig/appsCategorizes an issue or PR as relevant to SIG Apps.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions