Changelog since v1.27.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required for the custom scheduler plugin developers.
  Here's the breaking change in EnqueueExtension in the scheduling framework.
  The EventsToRegister in EnqueueExtension changed the return value from ClusterEvent to ClusterEventWithHint. ClusterEventWithHint allows each plugin to filter out more useless events via the callback function named QueueingHintFn.
  When the scheduling queue receives a cluster event, before moving each Pod from unschedulable pod pool to activeQ/backoffQ, it will call QueueingHintFn of plugins that rejected each Pod in the previous scheduling cycle.
  Depending on the value returned from QueueingHintFn, the scheduling queue changes how it queues each Pod:

  • if more than one QueueingHintFn returns QueueImmediately, it queues Pod to activeQ.
  • If no QueueingHintFn returns QueueImmediately and more than one plugin returns QueueAfterBackoff, it queues Pod to backoffQ if Pod is backing off, or to activeQ if Pod's backoff has already finished.
  • If all QueueingHintFn return QueueSkip, it puts this pod back to the unschedulable pod pool

  Having appropriate QueueingHintFn contributes to reducing useless retries and thus improves the overall scheduler's performance.

  How can I migrate?

  For backward compatibility, nil QueueingHintFn is treated as always returning QueueAfterBackoff.
  So, if you want to just keep the existing behavior, you can register ClusterEventWithHint with no QueueingHintFn in it.
  But, registering appropriate QueueingHintFn is, of course, better from a scheduling performance perspective. (#118551, @sanposhiho) [SIG Node, Scheduling, Storage and Testing]

• CephFS volume plugin (kubernetes.io/cephfs) has been deprecated in this release and will be removed in a subsequent release. The alternative is to use the CephFS CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes cluster. (#118143, @humblec)

• Deprecated support for CSI migration of Ceph RBD volumes. Users who were relying on Kubernetes' ability
  to migrate to an out-of-tree storage driver should complete that migration before the support for it is removed. (#118303, @carlory)

• RBD volume plugin (kubernetes.io/rbd) has been deprecated in this release
  and will be removed in a subsequent release. Alternative is to use RBD CSI driver
  (https://github.com/ceph/ceph-csi/) in your Kubernetes Cluster. (#118552, @humblec)

Changes by Kind

Deprecation

• Changed kubectl version default output to be identical to what kubectl version --short printed,
  and removed --short flag entirely. (#116720, @soltysh)
• Kube-controller-manager deprecate --volume-host-cidr-denylist and --volume-host-allow-local-loopback flags. (#118128, @carlory) [SIG API Machinery, Apps, Network, Node, Storage and Testing]
• Kubelet: The --azure-container-registry-config flag has been deprecated and will be removed in a future release, please use --image-credential-provider-config and --image-credential-provider-bin-dir to setup acr credential provider instead. (#118596, @SataQiu) [SIG Node]
• Removed tracking annotation from validation and defaulting. (#117633, @kannon92)
• Removed withdrawn feature NetworkPolicyStatus. (#115843, @rikatz)
• The deprecated flag --lock-object-namespace and --lock-object-name have been removed from kube-scheduler. Please use --leader-elect-resource-namespace and --leader-elect-resource-name or ComponentConfig instead to configure those parameters. (#119130, @SataQiu) [SIG Scheduling]
• KMSv1 is deprecated and will only receive security updates going forward. Use KMSv2 instead. In a future release, Set --feature-gates=KMSv1=true to use the deprecated KMSv1 feature. (#119007, @aramase)

Clusters/zones

• dev/weekly - Bootstrap
• Playground - Upgrade in place
• ext-mon - Upgrade in place
• platform - Upgrade in place
• c2 - Upgrade in place