@hectorj2f did you look at Termination of Pods, particularly step 5?

cc @smarterclayton

We've hit this when looking at how to do image builds using a container. Ideally, we'd run a container in the pod to set the image state, then commit and push that image in a follow on. But it requires a more complex lifecycle than our current flow. We had concerns about adding post-containers or post stop hooks when we discussed init containers.

@smarterclayton Which are those concerns ?

With the current lifecycle either, you increase the container functionality to cleanup itself when calling the PreStop operation of the pod lifecycle (this is quite ugly when those operations aren't related to the container logic :/ ); OR you have to hack some pods to be scheduled once you run a delete operation to do the cleanup. Both alternative are a bit ugly in my opinion.

What is it the blocker of this new feature for the pod lifecycle ?

@smarterclayton I'd like to help if it is necessary to push this issue.

Lifecycle hooks for different objects (eg. Deployments) discussed elsewhere:
#140
#3585
#14512

Also custom strategies are related to this - having the ability to customize the lifecycle of X means you can add your own hooks anywhere you want in the process. See #33545 (comment) for more information.

cc: @mfojtik @soltysh

@kubernetes/sig-apps

It sounds like what you are looking for is analogous to an init cotainer except that it runs after the pod is shutdown. Would a feature like this, lets call it a de-init container, be suitable to your requirements?

I think that having something similar to init-containers to be launched when the pod is terminated. It could make sense.

I understand that adding destroy-containers for a deployment, might imply to add a new state in the workflow. If it is necessary I won't see the problem but I understand your concerns about it.

Pod
  Spec
    Containers
    - OnShutdown:
        Exec /bin/cleanup.sh
        (LifecycleHook)

On the other hand, if we add a hook when deleting a deployment, my main concern is to able to use containers that are different than the containers that run in the Pod of the deployment as an example. If so, I'd be happy with your approach but I believe with your solution. We aren't able to run any logic which is outside of the own container like it happens with the current Termination hooks.

Doing so, it'd require to:

• Add additional logic to the containers that run in the pods to be able to clean themselves and whatever their init-container did.
• Add a new hook postStop cause preStop might limit the cleanup operations to be triggered in multiple case scenarios.

I can imagine some complications around that area.

How are we going to solve this problem:

• user defines cleanup action (whatever we will call it)
• node with a pod is down, therefore pod will be not able to call cleanup action
• user want to delete pod from k8s system
  It's obvious that then pod hooks weren't be executed, but then we will have 2 different behavior of the same.

Second problem:

• user defines cleanup action
• it crashes
  • should we proceed with pod delete
  • or maybe instead retry cleanup action? (how many times?)
• what if cleanup can't succeed because of some conditions (delete -f ?)

cc kubernetes/community#483

I have basically the same requirement, but for DaemonSet: I work on Weave Net which installs itself on every node via a DaemonSet. The install creates some side-effects on each node - network virtual devices, etc. - that a user would like to clear down if they decide to uninstall.

Users variously expect this to happen via kubectl delete or kubeadm reset, but currently there is no hook to run some uninstall code on each node.

We can't do an uninstall on a simple pod stop or delete - this will happen when the software is being upgraded, and destroying the pod network is very bad UX for an upgrade. We really need to know that the entire DaemonSet is being deleted.

@pigmej makes some good points, but it seems to me that users would understand that the code made some efforts to clean up, rather than no effort at all at present.

@bboreham kubeadm reset can be run individually on nodes. It is a common cleanup command for both as per this, but this does not delete the network interfaces as one would expect. One thing I observed is even after doing the kubeadm reset on the node, the master listed it as running. I don't know if it was because of the un-deleted network.

master listed it as running. I don't know if it was because of the un-deleted network.

No. It is because the code to remove a node was removed in #42713, @cheburakshu

@bboreham Will the master still schedule the node since it sees it? In my experience it did and the pod was in a pending state. Is it a correct behaviour?

@cheburakshu I dare say it will; however that is off-topic for this issue. Per #42713 the system administrator has to remove the node. Suggest you open an issue against kubeadm if this doesn't suit.

I have basically the same requirement, but for DaemonSet: I work on Weave Net which installs itself on every node via a DaemonSet. The install creates some side-effects on each node - network virtual devices, etc. - that a user would like to clear down if they decide to uninstall.

I have the same requirement. CNI plugins typically install files (binaries & a config file in /etc/cni/net.d/) in the host filesystem when the managing pod is first started as a DaemonSet.

I'm doing the installing part using a PostStart hook. For the cleanup I'm deleting the files in a PreStop hook. I'm not 100% positive yet whether this works perfectly fine all the time as the volume unmounting seems to happen in parallel with the PreStop.

In case this is not guaranteed, a naive thought is to consider a PreUnmount hook which would run before volume unmounting takes places.

In case this is not guaranteed, a naive thought is to consider a PreUnmount hook which would run before volume unmounting takes places.

Confirmed that this is not guaranteed. If I sleep in the PreStop hook for 60 seconds and then attempt to delete files on mounted volumes they will not be deleted as the volume has already been unmounted.

Hi @bboreham, @tgraf , @cheburakshu, could you please take a look at this proposal and provide more feedback it appears to me that we could achieve these using deferContainers, this will not only allow us to isolate termination scripts from the main image but also provide more guarantee and control than preStop hooks.

@dhillipkumars I couldn't see how that would allow me to know that the entire DaemonSet is being deleted. Conversely I don't seem to need additional isolation.

@bboreham hmmm... you may probably need this one too to precisely know why the pod is being brought down.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

/remove-lifecycle stale
/lifecycle frozen

How about using a finalizer?

In the case of a Pod, it would need permission to write to its own object, which is unorthodox.

In the case of a DaemonSet the pods could all add themselves as finalizers to the DaemonSet, but that gets out of hand at scale. Add another pod to coordinate shutdown (an operator, if you like), it feels like it could work.

Any update? I have the similar requirement now. I want a hook to do something when a job is deleted (or a pod is terminated). I tried to use the preStop hook, but it didn't work when I try to delete my jobs in a cron-job.

this proposal kubernetes/enhancements#1995 seems to handling this use case, please review it and see if that is in the right direction as per this issue

@krmayankk can you point more precisely to how kubernetes/enhancements#1995 would notify when a pod, deployment, daemonset, etc., is being deleted?

I think it could be extended to cover the case described here, by defining system-triggered notifications, but all I can see in the proposal as it stands is user-triggered.

/kind feature

I don't think we're going to implement new "do this when a pod is deleted" hooks in the core system any time soon. kubernetes/enhancements#1995 is not what you want here, it's what bboreham described - user-triggered. If someone wants to think up a general event-integration, it should at least start as an out-of core implementation.