What happened?

We encountered an issue where four pods in our GKE cluster have been stuck in the Terminating state for over three days. Jobs created these pods have already been deleted. Despite our attempts to delete the pods using standard methods (e.g., kubectl delete --force --grace-period=0), they remain in this state.

Here is the pod status:

kubectl get pods
NAME                                                       READY   STATUS        RESTARTS   AGE
allocation-28910880-tzr87                       0/2     Terminating   0          3d
allocation-28912320-8mn4z                       0/2     Terminating   0          3d
allocation-28913760-4gsxh                       0/2     Terminating   0          3d
allocation-28916640-vwdj2                       0/2     Terminating   0          3d

Additional details:

• Each pod has the finalizer batch.kubernetes.io/job-tracking and a QoS class of Burstable.
• Attempts to remove the finalizers result in the following error:

  The Pod "allocation-event-sync-28912320-8mn4z" is invalid: spec.initContainers: Forbidden: pod updates may not add or remove containers

• One of the pods (allocation-28912320-8mn4z) was running on a node (gke-primary-pool-bd44533c-aezp) that has since been drained and deleted. However, the pod still references the deleted node while remaining in the Terminating state.

Attempts to resolve:

1. Force deletion with kubectl delete --force --grace-period=0 and via kubectl replace.
2. Deleting via the kubectl proxy API (both deletion and replacement attempts).
3. Draining and deleting the node where one of the pods was running.
4. Attempting to modify the finalizers or QoS class, resulting in the above error.

What did you expect to happen?

The pods should be removed from the cluster once the Job is deleted, or at least be force-deletable using standard methods like kubectl delete.

How can we reproduce it (as minimally and precisely as possible)?

We cannot reproduce this issue at the moment. Creating a Job via the CLI or a YAML file and then deleting it does not result in the same stuck Terminating state. This appears to be a specific edge case that we can't relate to.

Anything else we need to know?

The only thing we had in the cluster was that we had recently updated the nodes from version 1.27 to 1.28, but we think that was before the job deletion.

Kubernetes version

$ kubectl version
Client Version: v1.28.13-gke.600
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
NAME="Container-Optimized OS"
ID=cos
PRETTY_NAME="Container-Optimized OS from Google"
HOME_URL="https://cloud.google.com/container-optimized-os/docs"
BUG_REPORT_URL="https://cloud.google.com/container-optimized-os/docs/resources/support-policy#contact_us"
GOOGLE_CRASH_ID=Lakitu
GOOGLE_METRICS_PRODUCT_ID=26
KERNEL_COMMIT_ID=3e0971e1551e88a5a9e615c239f034fd9fd8a423
VERSION=109
VERSION_ID=109
BUILD_ID=17800.309.13
$ uname -a
Linux gke-primary-pool-bd44533c-138r 6.1.100+ #1 SMP PREEMPT_DYNAMIC Sat Aug 10 14:21:56 UTC 2024 x86_64 Intel(R) Xeon(R) CPU @ 2.20GHz GenuineIntel GNU/Linux

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)